Enhancement of upload and/or download performance based on client and/or server feedback information

ABSTRACT

A system for client-server web applications is disclosed. Operations commence upon opening a client-server session configurable to establish a full-duplex persistent network communications between a client device and a server, then receiving an indication to upload or download one or more files or objects over the full-duplex persistent network communications.

RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 61/950,695, filed on Mar. 10, 2014, titled“Momentum Accelerator for Increasing Upload Speeds Associated with BulkUploads”, and is also a continuation-in-part of U.S. application Ser.No. 14/293,685, filed on Jun. 2, 2014, titled “Enhancement of Uploadand/or Download Performance Based on Client and/or Server FeedbackInformation”, which is a Divisional of U.S. application Ser. No.13/969,474, filed on Aug. 16, 2013, titled “Enhancement of Upload and/orDownload Performance Based on Client and/or Server FeedbackInformation”, now issued as U.S. Pat. No. 8,745,267 on Jun. 3, 2014,which claims the benefit of U.S. Provisional Application Ser. No.61/684,781, filed on Aug. 19, 2012, titled “Client and/or ServerFeedback Information Enabled Real Time or Near Real Time Optimizationand Enhancement of Upload/Download Performance and User Experience”,U.S. Provisional Application Ser. No. 61/694,466, filed on Aug. 29,2012, titled “Optimizations for Client and/or Server FeedbackInformation Enabled Real Time or Near Real Time Enhancement ofUpload/Download Performance”, U.S. Provisional Application Ser. No.61/702,154, Sep. 17, 2012, titled “Optimizations for Client and/orServer Feedback Information Enabled Real Time or Near Real TimeEnhancement of Upload/Download Performance”, U.S. ProvisionalApplication Ser. No. 61/703,699, filed on Sep. 20, 2012, titled“Optimizations for Client and/or Server Feedback Information EnabledReal Time or Near Real Time Enhancement of Upload/Download Performance”.The contents of the aforementioned applications are hereby incorporatedby reference in their entirety.

FIELD

This disclosure relates to the field of client-server web applications,and more particularly to techniques for dynamically shaping networktraffic when using full-duplex communications.

BACKGROUND

Web server to client (and client to web server) communications over HTTPneed to perform well under a wide variety of conditions. Legacycommunication techniques (e.g., asynchronous javascript and XML (AJAX))as used in many web applications rely on a middleware server thatupdates a portion of a web page such that it is possible to update partsof a web page without reloading the whole page. This facility can beused for implementing HTTP GET and PUT calls. Unfortunately, performingan HTTP GET and PUT calls involves establishment and tear-down of aconnection in a half-duplex regime, thus introducing overhead for eachHTTP GET or PUT. The effects of establishment and tear-down of aconnection in a half-duplex regime is exacerbated as the overall numberof data packets increases (e.g., with transfers of large numbers ofsmaller files), since the overhead becomes more significant as apercentage of overall time spent to perform the data transfers.

The websocket communication protocol can be used to establish apersistent communication socket, and thus permit full-duplexcommunications that can be parallelized and/or throttled during acommunication session. Unfortunately, the websockets facility has noprovisions for determining how much to parallelize and/or how much tothrottle. Absent sufficient information needed to determine how much toparallelize and/or how much to throttle (e.g., throttling ability lostor diminished at the application level), the traffic flow in acommunication session can suffer due to (1) poor performance related tooverwhelming the nodes in the route between the source and destination(e.g., if a lot of outgoing messages are sent in an oversized burst ofoutgoing HTTP traffic), or (2) poor performance due to under-utilizationof network resources (e.g., when too few parallelized outgoing messagesare sent in an undersized burst of outgoing HTTP traffic). What isneeded is a way to shape outgoing HTTP traffic based on upstream routingperformance observations. Moreover, what is needed is a way tocontinuously shape outgoing HTTP traffic based on upstream routingperformance observations that may change during a communication session.

The problem to be solved is rooted in technological limitations oflegacy approaches. Improved techniques, in particular improvedapplication of technology, are needed to address the problem ofimproving the performance of uploads and downloads of content over HTTP.More specifically, the technologies applied in the aforementioned legacyapproaches fail to achieve the sought-after capabilities of theherein-disclosed techniques for dynamically shaping network traffic whenusing full-duplex communications.

SUMMARY

The present disclosure provides improved systems, methods, and computerprogram products suited to address the aforementioned issues with legacyapproaches. More specifically, the present disclosure provides adetailed description of techniques used in systems, methods, and incomputer program products for dynamically shaping network traffic whenusing a full-duplex communications channel (e.g., over websockets). Thevarious embodiments address the problem of performance of uploads anddownloads of content over HTTP, which is often stochastic, leading topoor and/or unpredictable performance and a poor or unpredictable userexperience. Certain embodiments are directed to technological solutionsfor shaping outgoing HTTP traffic based on upstream routing performanceobservations, as well as advancing peripheral technical fields. Thedisclosed embodiments modify and improve over conventional approaches.In particular, practice of the disclosed techniques improves performanceeven while using the same amount of computer processing power, andreduces communication overhead needed for dynamically shaping networktraffic when using full-duplex communications. Some embodimentsdisclosed herein use techniques to improve the functioning of multiplesystems within the disclosed environments.

One embodiment implements a method for using feedback information todynamically adapt to changes in performance characteristics of a datatransfer. The method commences upon opening a client-server sessionconfigurable to establish full-duplex persistent network communications(e.g., using a channel or socket) between a client device and a server,then receiving an indication to upload a file or object over thefull-duplex persistent network communication socket from a locationaccessible by the client device to one or more storage locationsaccessible by the server. The server allocates resources to open thefull-duplex persistent network communication socket, which in turn isconfigured for receiving, from the client device, an initial portion ofa data transfer stream comprising a succession of portions of the fileor object. The server selects a first host from a group of hosts to usefor transferring at least some of the succession of portions of the fileor object. Upon receiving a set of performance characteristicspertaining to the data transfer stream, the server determines one ormore traffic shaping parameters.

Further details of aspects, objectives, and advantages of the disclosureare described below and in the detailed description, drawings, andclaims. Both the foregoing general description of the background and thefollowing detailed description are exemplary and explanatory, and arenot intended to be limiting as to the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described below are for illustration purposes only. Thedrawings are not intended to limit the scope of the present disclosure.

FIG. 1A1 presents an environment having host servers in a host farmwhere the servers are configurable to be capable of using networkpathway feedback information for dynamically shaping network trafficwhen using full-duplex communications, according to an embodiment.

FIG. 1A2 presents an environment to show half-duplex communicationthrough an intermediate node to process HTTP requests.

FIG. 1A3 presents an environment showing full-duplex communicationthrough an intermediate node to process websocket requests whendynamically shaping network traffic, according to an embodiment.

FIG. 1B depicts variations of progress bars to present to a user whenuploading files over websockets, according to an embodiment.

FIG. 1C1 presents a timeline to show unnecessary and/or adverse latencyeffects when using a half-duplex communication mechanism.

FIG. 1C2 presents a timeline to show reduced latency when using afull-duplex communication mechanism in an environment configured todynamically route uploaded files along the fastest paths, according tosome embodiments.

FIG. 1D1 depicts a set of websocket frames.

FIG. 1D2 depicts a websocket frame prepending technique to includeinformation used at nodes along a pathway for dynamically shapingnetwork traffic when using full-duplex websocket communications,according to an embodiment.

FIG. 1E1 depicts a series of response blocks that carry acknowledgementinformation.

FIG. 1E2 depicts a series of response blocks that carrydynamically-determined pathway information for use when adaptivelyshaping network traffic during full-duplex websocket communications,according to some embodiments.

FIG. 1F is a chart showing a technique for dynamically determining aclient-side shaping parameter based on values within a response oracknowledgement, according to some embodiments.

FIG. 2 is a diagram of a web-based or online collaboration platformdeployed in an enterprise or other organizational setting for organizingwork items and workspaces, accessible using a mobile site or using amobile platform, according to some embodiments.

FIG. 3 is a diagram of a system having a host server, origin server orcontent server capable of using feedback information to select an upload(or download) pathway from a number of alternative pathways, accordingto some embodiments.

FIG. 4 is a diagram of a feedback loop for optimizing or enhancingupload (or download) performance, according to some embodiments.

FIG. 5A presents a diagram of a feedback loop for optimizing orenhancing upload (or download) performance, according to someembodiments.

FIG. 5B presents a block diagram showing instances of selectedcomponents of a feedback loop optimizer, according to some embodiments.

FIG. 6A is a block diagram depicting example components of a user devicethat provides some of the feedback information that is used by a hostserver to select a pathway to upload/download files to optimize orenhance upload and/or download performance, according to someembodiments.

FIG. 6B is a block diagram showing example components of a client-sidefeedback loop optimizer used for dynamically shaping network trafficwhen using full-duplex websocket communications, according to someembodiments.

FIG. 7A is a block diagram depicting example components of a sync clientthat can use feedback information to select a pathway to optimize orenhance upload performance by dynamically shaping network traffic,according to some embodiments.

FIG. 7B is a block diagram depicting example components of a sync serverthat communicates with a sync client, according to some embodiments.

FIG. 8 is a flow diagram depicting a method implemented by a host serverfor recommending an upload pathway for a given set of characteristics toachieve enhanced upload performance by dynamically shaping networktraffic, according to some embodiments.

FIG. 9A is a logic flow diagram depicting a method for selecting apathway from an array of pathways based on a set of weights, accordingto some embodiments.

FIG. 9B is a logic flow diagram depicting a method for selecting apathway from an array of pathways based on a set of weights cached in auser device, according to some embodiments.

FIG. 10 is a logic flow diagram depicting a method for selecting anoptimal upload pathway by a user device based on client-side feedbackinformation, according to some embodiments.

FIG. 11 is a logic flow diagram depicting a method for using feedbackinformation to optimize data transfer performance, according to someembodiments.

FIG. 12 is a logic flow diagram depicting a method for optimizing filetransfer performance to improve user experience, according to someembodiments.

FIG. 13 is a logic flow diagram depicting a method for enhancing theupload performance, according to some embodiments.

FIG. 14 is a chart depicting uploads by pathways over time as generatedusing information tracked or observed at the host server, according tosome embodiments.

FIG. 15 is a system using pathway feedback information to optimize orenhance full-duplex upload or download performance, according to someembodiments.

FIG. 16 depicts a system as an arrangement of computing modules that areinterconnected so as to operate cooperatively to implement certain ofthe herein-disclosed embodiments.

FIG. 17 depicts an exemplary architecture of components suitable forimplementing embodiments of the present disclosure, and/or for use inthe herein-described environments.

DETAILED DESCRIPTION

Some embodiments of the present disclosure address the problem ofperformance of uploads and downloads of content over HTTP. HTTP GET andPUT calls often perform stochastically, leading to poor performance ofuploads and downloads. Some embodiments are directed to approaches forshaping outgoing HTTP traffic based on upstream routing performanceobservations. More particularly, disclosed herein and in theaccompanying figures are exemplary environments, systems, methods, andcomputer program products for dynamically shaping network traffic whenusing full-duplex communications (e.g. using websockets).

Overview

Legacy implementations (e.g., AJAX) rely on half-duplex communicationsbetween a client and its connected server. AJAX POSTs use two packets:(1) a header packet is sent first, (2) once the server acknowledges thesuccessful receipt of the header, then (3) the client sends the body ofthe POST request. In contrast, websockets support the establishment of apersistent socket, and once the socket has been successfullyestablished, any number of websocket data frames can be sentasynchronously between the client and the server in full-duplex mode.Further, both text and binary frames can be sent in full-duplex mode(e.g., in either direction, and at the same time). Text payload isframed with just two bytes when using websockets: each frame starts witha 0x00 byte and ends with a 0xFF byte (with UTF-8 data in between).Websocket binary frames use a length prefix.

Various characteristics of half-duplex communication as compared withfull-duplex communication are shown and discussed herein, and any of thetechniques disclosed herein can be used in a wide variety ofcollaboration environments.

Various embodiments are described herein with reference to the figures.It should be noted that the figures are not necessarily drawn to scaleand that the elements of similar structures or functions are sometimesrepresented by like reference numerals throughout the figures. It shouldalso be noted that the figures are only intended to facilitate thedescription of the disclosed embodiments—they are not representative ofan exhaustive treatment of all possible embodiments, and they are notintended to impute any limitation as to the scope of the claims. Inaddition, an illustrated embodiment need not portray all aspects oradvantages of usage in any particular environment. An aspect or anadvantage described in conjunction with a particular embodiment is notnecessarily limited to that embodiment and can be practiced in any otherembodiments even if not so illustrated. Also, reference throughout thisspecification to “some embodiments” or “other embodiments” means that aparticular feature, structure, material, or characteristic described inconnection with the embodiments is included in at least one embodiment.Thus, the appearances of the phrase “in some embodiments” or “in otherembodiments” in various places throughout this specification are notnecessarily referring to the same embodiment or embodiments.

DEFINITIONS

Some of the terms used in this description are defined below for easyreference. The presented terms and their respective definitions are notrigidly restricted to these definitions—a term may be further defined bythe term's use within this disclosure. The term “exemplary” is usedherein to mean serving as an example, instance, or illustration. Anyaspect or design described herein as “exemplary” is not necessarily tobe construed as preferred or advantageous over other aspects or designs.Rather, use of the word exemplary is intended to present concepts in aconcrete fashion. As used in this application and the appended claims,the term “or” is intended to mean an inclusive “or” rather than anexclusive “or”. That is, unless specified otherwise, or is clear fromthe context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A, X employs B, or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances. The articles “a” and “an” as used in thisapplication and the appended claims should generally be construed tomean “one or more” unless specified otherwise or is clear from thecontext to be directed to a singular form.

Reference is now made in detail to certain embodiments. The disclosedembodiments are not intended to be limiting of the claims.

Descriptions of Exemplary Embodiments

FIG. 1A1 presents an environment 1A100 having host servers in a hostfarm where the servers are configurable to be capable of using networkpathway feedback information for dynamically shaping network trafficwhen using full-duplex websocket communications. As an option, one ormore instances of environment 1A100 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the environment 1A100 or any aspectthereof may be implemented in any desired environment.

As shown, a set of users (e.g., collaborators 118) use various userdevices 102 to interact with one or more workspaces 122. The workspacescan be stored in any location, and are at least partially maintained bycomponents within a host farm 101. The host farm 101 supports anyvariety of processing elements and/or storage devices and/or serverssuch as a host server 115, a sync server 120, a notification server 150,a collaboration server, a content server, an origin server, etc. In somecases the host farm is implemented in whole or in part by a cloud serverplatform and/or a cloud storage platform.

The user can edit the file accessed from the host farm 101 without theadditional process of manually downloading and storing the file locallyon the user device 102. For example, the file may be ready for the userto edit locally, even without informing the user where the file isstored or without prompting the user for a directory in which to storethe file. Such a facility streamlines the frequently repeated accessingand editing processes—and enhance the user experience.

Functions and techniques performed by the host farm 101, the client sidecomponents on a user device 102, a sync client on a user device 102, async server 120 and other related components therein are described,respectively, in detail with further reference to the examples.

Users (e.g., collaborators 118) use such cloud server platform(s) and/orcloud storage platform(s) to share files of all sizes and from anylocation to any other location, however near or far. Sharing activitiesoften include uploads 127 and downloads 129, and sometime includeuploads and downloads of large files or other objects. In practice,upload and/or download speeds tend to decrease as users are physicallyfarther away from the location of data centers for the collaboration orcloud storage platforms. Even with third-party content delivery networks(e.g., Akamai) to accelerate uploads/downloads, some users cannotachieve speeds anywhere near what their Internet connection is able tosupport. Furthermore, sometimes direct upload can be much faster than anupload through an accelerator such as those provided by content deliverynetworks. The disclosed technology solves these problems by usingnetwork optimization and routing strategies to enhance data transferperformance and improve user experience.

Embodiments of the present disclosure include systems and methods forenhancing or optimizing upload and/or download performance usingfeedback information. The feedback information is collected by aclient-side and/or server-side feedback loop that can be used toappropriately gauge the speed for individual locations, browsers and/oroperating systems and select an upload/download pathway to obtain realtime or near real time enhancement and optimization of upload/downloadperformance (e.g., speed, reliability) and improvement in userexperience.

Further optimization for uploading/downloading larger files can also beimplemented,—in particular, use of full-duplex communication togetherwith information observed over a network route,—can be used to greatlyincrease the upload and download performance. In exemplary embodiments,unnecessary latency introduced by the legacy half-duplex communicationschemes is reduced or eliminated by using full-duplex communicationtechniques. Several comparisons between half-duplex communication andfull-duplex communications use for uploading and downloading are shownand discussed in the following FIG. 1A2 and FIG. 1A3.

FIG. 1A2 presents an environment 1A200 to show half-duplex communicationthrough an intermediate node to process HTTP requests. As an option, oneor more instances of environment 1A200 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the environment 1A200 or any aspectthereof may be implemented in any desired environment.

As shown, a collaborator interacts with a user device while performingcollaboration activities such as browser activities 154. During thecourse of performing such activities, the collaborator may wish toupload a file, and may so indicate as much (e.g., through clicks orpresses, or swipes, etc.). A portion of a web application that executeson the user device initiates the shown half-duplex protocol 121. In thisexample of half-duplex communication, the half-duplex protocol begins bysending a header (see message 156 ₁), and then the web application waitsfor an acknowledgement. In some situations the header is received bymiddleware (e.g., intermediate node 112 or edge node 104) which willprocess the header (see message 158 ₁) and forwards it on to the shownserver (see send header message 156 ₂). The status that the header hasbeen received is communicated to the server (see message 160 ₁), and theclient will commence to send payload corresponding to the header (seemessage 162 ₁). Additional browsing activities can then proceed.

This half-duplex protocol 121 has many variations such as including orexcluding middleware components. Regardless of such variations, thecharacteristics of communications in a half-duplex communication mode152 introduces latency, and in some cases a large amount of latency, andin most cases unnecessary latency. Strictly as one example,implementations of web application components may implement a statetransition regime where the intermediate node (or server node) opens asocket upon an HTTP call, processes the HTTP call, and then closes thesocket to release resources. In situations or applications where a largenumber of HTTP calls need to be executed (e.g., for an upload of a largefile), or when real-time performance is needed (e.g., forfrequently-updated communications between client and server) theestablishment and tear-down of a socket for each transmission iswasteful, increases latency, and serves to undesirably reduce and/orlimit total available throughput. What is needed in such situations orapplication is a full-duplex communication regime, such as is shown anddiscussed as pertaining to FIG. 1A3.

FIG. 1A3 presents an environment 1A300 showing full-duplex communicationthrough an intermediate node to process websocket requests whendynamically shaping network traffic. As an option, one or more instancesof environment 1A300 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the environment 1A300 or any aspect thereof maybe implemented in any desired environment.

The shown full-duplex protocol 131 uses a full-duplex communication mode168 to process websocket requests and receive websocket responses. Thefull-duplex protocol includes opening a socket (see message 170), whichoperation allocates resources to use and maintain for ongoingcommunications over the socket, then receiving a confirmation that thesocket is open (see message 172), and then sending requests andreceiving responses asynchronously. In the example shown, many requests(e.g., portions, packets or parcels, etc.) are sent in succession (e.g.,P1, P2, P3, P4, P5, P6). The many requests are sent in successionwithout waiting for corresponding acknowledgements for each of theindividual requests. This has the effect of reducing latency since the‘next’ portion can be sent without waiting for an acknowledgement of the‘previous’ portion. In the example, many portions (see P1, P2, P3, P4,P5, P6) are sent before the acknowledgement (see message ACK 174 ₁) ofthe ‘first’ packet (see R1). As shown, many next portions are sent insuccession (e.g., P7, P8, P9, P10, P11, P12)—even before receiving theacknowledgement (see message ACK 174 ₂) of the ‘next’ packet (see R2).

Full-duplex communication has many desirable performancecharacteristics, however since any number of packets or portions can besent without waiting for an acknowledgement of the ‘previous’ portion orportions, such a communication scheme can over-demand network resourcesin the route from the source to the destination. Various embodimentsserve to dynamically shape network traffic so as to avoid overwhelmingcommunication components along the route from the source to thedestination.

Dynamic shaping of full-duplex communications from the source to thedestination serves not only to optimize marshalling of networkresources, but also to facilitate a predictable experience for the user.More specifically, and as discuss hereunder, the user can see (e.g.,using a progress bar or other screen device) the progress of thetransmission(s) such as an upload of a large file. Selectedimplementations of progress bars are discussed as follows.

FIG. 1B depicts variations of progress bars 1B00 to present to a userwhen uploading files over websockets. As an option, one or moreinstances of progress bars 1B00 or any aspect thereof may be implementedin the context of the architecture and functionality of the embodimentsdescribed herein. Also, the progress bars 1B00 or any aspect thereof maybe implemented in any desired environment.

In one embodiment, a progress bar indicating upload progress of theupload request is depicted in the user interface. The progress bartypically indicates the progress of the upload of the full request. Forexample, if the request is a multi-file upload request, the progress barindicates the progress of uploading all of the files. In addition, theprogress bar can further indicate the total size of the upload, timeelapsed, time remaining, average speed of upload, completed upload filesize, and/or total size or number of files that have completed. Uploadprogress can be determined since at any moment the uploader knows thetotal bytes that have been transferred, the time elapsed, and the totalsize of the upload. In one embodiment, the progress bar is depicted evenwhen the user navigates away from the user interface to another userinterface during the upload process.

In the example, alongside “Time=0000” an unpopulated progress bar 130 isdisplayed. At “Time=0010” the partially complete progress bar 132indicates that a series of portions have been sent, and that someresponses have been received. At time “Time=0020” the almost completedprogress bar 134 is filled out to its full length, showing a segmentremaining incomplete (see the “assembling” label).

Such a model of a progress bar for capturing progress duringasynchronous communication can be decorated with additional informationencoded into shapes and/or colors. For example, the sent-encodedprogress bar 136 can indicate (e.g., by a color coding or othergraphical treatment) that a set of portions have been sent. Thisprogress bar can be updated in real time to indicate that a particularsent portion has been acknowledged. The sent-acknowledged progress bar138 depicts that portions P1, P2, P3, P4, P5, and P6 have been sent andacknowledges, and portions P7 and P8 have been sent and are awaitingacknowledgement. The order of acknowledgements received and indicatedneed not be the same order as the order that the sent portions weresent.

When all portions have been sent and their correspondingacknowledgements received, the assembling progress bar 142 indicatesthat a second task is taking place. In some cases, when all portionshave been sent and all except one of their correspondingacknowledgements have been received, the assembling progress bar 142indicates that a second task is taking place. The assembling screendevice and label can indicate the progress of the second task, forexample, by shading or color coding to indicate the ratio of thecompleted last steps portion 144 as compared to the in-progress laststeps portion 148.

When using full-duplex communication, characteristics of the progressbar can be presented to the user to indicate an estimated total time.Techniques for estimation are discussed as pertaining to the followingcomparison timelines.

FIG. 1C1 presents a timeline 1C100 to show unnecessary latency effectswhen using a half-duplex communication mechanism. As an option, one ormore instances of timeline 1C100 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the timeline 1C100 or any aspectthereof may be implemented in any desired environment.

When using a half-duplex communication mechanism, the total timeestimated for an upload (e.g., an upload of a large file) can becalculated using the formula:

Time=N×(t_start+t_upload+t_response)  (EQ. 1)

An estimate based on such a calculation is dependent on the timeduration of the first start-transfer-response sequence. As shown thelatency for the three transfers consumes three units of time.

FIG. 1C2 presents a timeline 1C200 to show reduced latency when using afull-duplex communication mechanism in an environment configured todynamically route uploaded files along the fastest paths. As an option,one or more instances of timeline 1C200 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the timeline 1C200 or any aspectthereof may be implemented in any desired environment.

When using a full-duplex communication mechanism, the total timeestimated for an upload (e.g., an upload of a large file) can becalculated using the formula:

Time=1×t_start+(N×t_upload)+1×t_response  (EQ. 2)

An estimate based on such a calculation is dependent on the timeduration of the first start-transfer-response sequence. As shown thelatency for the three transfers consumes far less than three unitsdiscussed in FIG. 1C1.

The shown start block 1C202 can comprise only a very small amount ofinformation, in some cases only the information needed to open apersistent socket. In other cases the start block 1C202 includes anidentification of a system or server to be the recipient of the uploadeddata. In other situations, the start block 1C202 includes anidentification of an upload system and/or information pertaining to aparticular route or pathway, and/or indication of service levelagreement-related parameters such as dedicated bandwidth by time ordate, reservations for isochronous bandwidth, etc.

The estimate of time remaining can be continuously recalculated based onthen-current network performance. In some cases, and as shown anddiscussed as pertaining to FIG. 1D2, a packet comprising a preamble canbe sent ahead of a following packet, and the preamble can provideinformation that can in turn be used by intermediate nodes to manage thecapture and sending of performance measurements. One possible use ofsuch a preamble is to juxtapose the preamble with a respective websocketframe.

FIG. 1D1 depicts a set of websocket frames 1D100. As an option, one ormore instances of websocket frames 1D100 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the websocket frames 1D100 or anyaspect thereof may be implemented in any desired environment.

The embodiment shown in FIG. 1D1 is merely one example of a series ofwebsocket frames. As earlier shown (e.g., see FIG. 1A3) websocketssupport the establishment of a persistent socket, and once the sockethas been successfully established, any number of websocket data framescan be sent asynchronously between the client and the server usingfull-duplex modes. Further, both text and binary frames can be sentfull-duplex (e.g., in either direction, and at the same time). Usingwebsockets, the payload is framed with framing bytes: in the case oftext frames, each frame starts with a 0x00 byte, and ends with a 0xFFbyte (with UTF-8 data in between). Websocket text frames use aterminator, and binary frames use a length prefix. Frames within aseries can be sent in any order and/or use any mixture of text andbinary frames. The example of FIG. 1D1 shows text frames 176 (e.g., textframe 176 ₁, text frame 176 ₂) and a binary frame 178 (e.g., binaryframe 178 and text frame 176 ₃). These frames can be sent in any orderas may be deemed by the sender. A text frame can be preceded (orfollowed) by a text frame or a binary frame, and the contents of thepreceding (or following) frame can be application dependent. One exampleof information used by network components that may be encountered alonga pathway from sender to receiver and back is given as shown anddescribed as pertains to FIG. 1D2.

FIG. 1D2 depicts a websocket frame prepending technique 1D200 to includeinformation used at nodes along a pathway for dynamically shapingnetwork traffic when using full-duplex websocket communications. As anoption, one or more instances of websocket frame prepending technique1D200 or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the websocket frame prepending technique 1D200 or any aspectthereof may be implemented in any desired environment.

The example shown in FIG. 1D2 include a preamble 180 in the form of awebsocket frame. As shown, the contents include a name (e.g., a name ofa file or other object such as an object in volatile memory), a size(e.g., the size of the object), and a target (e.g., repository name orlocation).

In some situations, a text frame (or binary frame) is preceded by apreamble 180 that serves to contain information pertaining to afollowing (or preceding) frame. Such a preceding (or following) framecan include metadata pertaining to the upload file or object, and/orinformation used by network components that may be encountered along apathway from sender to receiver and back. For example, the metadata caninclude a filename or object identifier, a size (e.g., in bytes), thesensitivity of the file or object, an indication of the owner or accountthat holds the file or object, and/or any aspects of importance,priority, retention policy, and/or any other storage-related policy orservice level agreement. In one case, an ownership record or policy orservice level agreement might be used to determine aspects of the targetstorage. For example, responsive to a characteristic of a policy or aservice level agreement, the uploaded file might be stored in “Tier-1”(fast access) storage, or might be stored at a Tier-N (archive access)storage location, and/or a policy or service level agreement mightindicate that a file is to be replicated over multiple locations (e.g.,for disaster recovery) or mirrors (e.g., for fast access across widegeographies).

A preamble 180 can be sent once per upload session (or download session)or can be sent once per file (e.g., in a multiple-file or batch transfersession), or can be sent periodically or at any time during thepersistence period of a socket.

A preamble 180 can be constructed so as to provide the information(e.g., identifiers, lookup keys, etc.) that might be needed by thereceiving server. Such information can be organized into one packet, orcan be organized to span multiple packets.

FIG. 1E1 depicts a series of response blocks 1E100 that carryacknowledgement information. As an option, one or more instances ofresponse blocks 1E100 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the response blocks 1E100 or any aspect thereofmay be implemented in any desired environment.

The shown websocket frames (e.g., response R1 184, response R2 186,response R3 188) comprise acknowledgements, however the frame cancomprise other information as may be deposited by nodes along a pathway(e.g., node engaged in full-duplex websocket communications). Suchinformation deposited by nodes along a pathway can be used fordynamically shaping network traffic.

FIG. 1E2 depicts a series of response blocks 1E200 that carrydynamically-determined pathway information for use when adaptivelyshaping network traffic during full-duplex websocket communications. Asan option, one or more instances of response blocks 1E200 or any aspectthereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the responseblocks 1E200 or any aspect thereof may be implemented in any desiredenvironment.

The shown websocket frames (e.g., response R1′ 190, response R2′ 192,response R3′ 194) comprise the semantics of acknowledgements, howeverthe frame can comprise pathway information as may be deposited by nodesalong a pathway (e.g., a node engaged in full-duplex websocketcommunications). Such information deposited by nodes along a pathway canbe used for dynamically shaping network traffic. As traffic flows alonga pathway, performance information may be deposited (e.g., into aresponse frame) by nodes along the pathway, and the deposit or insertionof such pathway performance information can take place continuously asresponse packets are processed by nodes along the pathway. As trafficpatterns change (e.g., as there is congestion, or as additionalresources become free, or as better routes are identified) theperformance information as may be deposited by nodes along the pathwaycan be updated, and such updated information can be used by a sender toshape outgoing traffic so as to avoid overwhelming resources on thepathway while still sending websocket frames as fast as the network ispredicted to be able to process the traffic. There are many parametersuseful to form predictions and to dynamically adapt or shape trafficbased on the then-current estimates, which are based at least in part onupdates to the performance information that is continuously deposited bynodes along the pathway. One dynamic traffic-shaping technique relies onmanaging a number of inflight requests (e.g., request that have not yetbeen acknowledged). Use of such a technique is shown and discussed aspertains to FIG. 1F.

FIG. 1F is a chart 1F00 showing a technique for dynamically determininga client-side shaping parameter based on values within a response oracknowledgement. As an option, one or more instances of chart 1 F00 orany aspect thereof may be implemented in the context of the architectureand functionality of the embodiments described herein. Also, the chart 1F00 or any aspect thereof may be implemented in any desired environment.

The chart 1F00 depicts a shape formed by an initial point 193, anupdated (lowered) point 195, and the newer updated (increased) point197. In some cases the difference between one or more latencypredictions and respective one or more actual measured predictions canbe used to shape traffic. In some cases, a websocket response is used todynamically determining a number of inflight requests. In other casesother parameter are used. Strictly as examples, the shaping parametercan include one or more of, an inter-packet gap period, an encodingtechnique or encoding parameter, a lower limit, and upper limit, etc.

Exemplary Use Models

The user devices 102 can be any system and/or device and/or anycombination of devices/systems that is able to establish a connectionincluding wired, wireless, cellular connections with another device, aserver and/or other systems such as the host farm 101, a sync server 120and/or notification server 150 directly or via hosts or pathways such asone or more middleware nodes or networks (e.g., content deliverynetworks 110 (CDNs)), and/or such as one or more network edge nodes(e.g., distributed edge nodes 104). The user devices 102 will typicallyinclude a display and/or other output functionality(ies) to presentinformation and data exchanged between/among the user devices 102 and/orthe host farm 101, the sync server 120 and/or the notification server150, and optionally the distributed edge nodes 104 and the third-partycontent delivery networks 110.

For example, the user devices 102 can include mobile, hand-held orportable devices or non-portable devices and can be any of, but notlimited to, a server desktop, a desktop computer, a computer cluster, orportable devices including a notebook, a laptop computer, a handheldcomputer, a palmtop computer, a mobile phone, a cell phone, a smartphone, a PDA, a Blackberry device, a Treo, a handheld tablet (e.g., aniPad, a Galaxy, Xoom Tablet, etc.), a tablet PC, a thin-client, a handheld console, a hand held gaming device or console, an iPhone, and/orany other portable, mobile, hand held device, etc, running on anyplatform or any operating system (e.g., Mac-based OS (OS X, iOS, etc.),Windows-based OS (Windows Mobile, Windows 7, etc.), Android, BlackberryOS, Embedded Linux platforms, Palm OS, Symbian platform, etc.). In oneembodiment, the user devices 102, the host farm 101, the notificationserver 150, the distributed edge nodes 104, and/or the content deliverynetworks 110 are coupled via a network 106. In some embodiments, theuser devices 102 and the host farm 101 may be directly connected to oneanother.

The input mechanism on the user devices 102 can include a touch screenkeypad (including single touch, multi-touch, gesture sensing in 2D or3D, etc.), a physical keypad, a mouse, a pointer, a track pad, a motiondetector (e.g., including 1-axis, 2-axis, or 3-axis accelerometer), alight sensor, a capacitance sensor, a resistance sensor, a temperaturesensor, a proximity sensor, a piezoelectric device, a device-orientationdetector (e.g., electronic compass, tilt sensor, rotation sensor,gyroscope, accelerometer, etc.), or any combination of the above.

Signals received or detected indicating user activity at the userdevices 102 through one or more of the above input mechanisms, orothers, can be used in the disclosed technology by various users orcollaborators 118 (e.g., user collaborator 123, admin collaborator 124,creator collaborator 125) for accessing, through network 106, aweb-based collaboration environment or online collaboration platform ora cloud-based storage platform (e.g., hosted by the host farm 101).

The online collaboration platform or a cloud-based storage platform isused by users to share files of all sizes with others around the world.Users can upload/download files from physical datacenters. Depending onthe location, a user may be located closer or farther away from aphysical datacenter where files are stored. Since upload/download speedstend to decrease as users become farther away from physical datacenters,hosts or intermediary nodes (e.g., a file uploader/downloader) can beused to accelerate uploads/downloads.

The cloud-based storage platform can store work items uploaded by users,and can allow download of work items by users. The collaborationplatform or environment hosts workspaces 122 with work items that one ormore users can access (e.g., view, edit, update, revise, comment,download, preview, tag, or otherwise manipulate). A work item cangenerally include any type of digital or electronic content that can beviewed or accessed via an electronic device (e.g., user device 102). Thedigital content can include .pdf files, .doc files, slides (e.g.,PowerPoint slides), images, audio files, multimedia content, web pages,blogs, etc. A workspace can generally refer to any grouping of a set ofdigital content in the collaboration platform. The grouping can becreated, identified, or specified by a user or through other means. Thisuser may be a creator user or an administrative user, for example.

In general, a workspace can be associated with a set of users orcollaborators (e.g., collaborators 118) who have access to the contentincluded therein. The levels of access (e.g., based on permissions orrules) of each user or collaborator to access the content in a givenworkspace may be the same or may vary among the users. Each user mayhave their own set of access rights to every piece of content in theworkspace, or each user may have different access rights to differentpieces of content. Access rights may be specified by a user associatedwith a workspace and/or a user who created/uploaded a particular pieceof content to the workspace, or any other designated user orcollaborator.

In general, the collaboration platform allows multiple users orcollaborators to access or collaborate efforts on work items such thateach user can see edits, revisions, comments, or annotations being maderemotely to specific work items through their own user devices 102. Forexample, a user can upload a document to a workspace for other users toaccess (e.g., for viewing, editing, commenting, signing-off, orotherwise manipulating). The user can login to the online platform andupload the document (or any other type of work item) to an existingworkspace or to a new workspace. The document can be shared withexisting users or collaborators in a workspace.

A diagrammatic illustration of the online collaboration environment andthe relationships between workspaces and users/collaborators areillustrated with further reference to the example of FIG. 2.

FIG. 2 is a diagram 200 of a web-based or online collaboration platformdeployed in an enterprise or other organizational setting for organizingwork items and workspaces, accessible using a mobile site or using amobile platform. As an option, one or more instances of diagram 200 orany aspect thereof may be implemented in the context of the architectureand functionality of the embodiments described herein. Also, the diagram200 or any aspect thereof may be implemented in any desired environment.This figure shows an example diagram of a web-based or onlinecollaboration platform deployed in an enterprise or other organizationalsetting 250 for organizing work items and workspaces 205, 225, and 245,accessible using a mobile site or using a mobile platform.

The web-based platform for collaborating on projects or jointly workingon documents can be used by individual users and shared amongcollaborators. In addition, the collaboration platform can be deployedin an organized setting including, but not limited to, a company (e.g.,an enterprise setting), a department in a company, an academicinstitution, a department in an academic institution, a class or coursesetting, or any other types of organizations or organized setting.

When deployed in an organizational setting, multiple workspaces (e.g.,workspace A, B C) can be created to support different projects or avariety of work flows. Each workspace can have its own associate workitems. For example, work space A 205 may be associated with work items215, work space B 225 can be associated with work items 235, and workspace N 245 can be associated with work items 255. The work items 215,235, and 255 may be unique to each work space but need not be. Forexample, a particular word document can be associated with only one workspace (e.g., work space A 205) or it may be associated with multiplework spaces (e.g., work space A 205 and work space B 225).

In general, each work space has a set of users or collaboratorsassociated with it. For example, work space A 205 is associated withmultiple users or collaborators 206. In some instances, work spacesdeployed in an enterprise may be department specific. For example, workspace B may be associated with department 210 and some users shown asexample user A 208 and workspace N 245 can be associated withdepartments 212 and 216 and users shown as example user B 214.

Each user associated with a work space can generally access the workitems associated with the work space. The level of access will depend onpermissions associated with the specific work space, and/or with aspecific work item. Permissions can be set for the work space or setindividually on a per work item basis. For example, the creator of awork space (e.g., one of user A 208 who creates work space B) can set apermission setting applicable to all work items 235 for other associatedusers and/or users associated with the affiliate department 210. Creatoruser A 208 may also set different permission settings for each workitem, which may be the same for different users, or varying fordifferent users.

In each work space A, B . . . N, when an action is performed on a workitem by a given user or any other activity is detected in the workspace, other users in the same work space may be notified (e.g., in realtime or in near real time, or not in real time). Activities whichtrigger real time notifications can include, by way of example but notlimitation, adding, deleting, or modifying collaborators in the workspace, uploading, downloading, adding, deleting a work item in the workspace, creating a discussion topic in the work space.

Specifically, items or content downloaded or edited in accordance withthe techniques described in the present disclosure can causenotifications to be generated. Such notifications can be sent torelevant users to notify them of actions surrounding a download, anedit, a change, a modification, a new file, a conflicting version, or anupload of an edited or modified file.

A workspace in an online or web-based collaboration environment isaccessible by multiple collaborators through various devices, includingvia a mobile site or mobile platform associated with the collaborationenvironment.

Each user can individually use multiple different devices to accessand/or manipulate work items in a work space with which they areassociated with. For example, users can be collaborators on a project towhich certain work items are relevant. Since the work items are hostedby the collaboration environment (e.g., a cloud-based environment), eachuser can access the work items anytime, and from any physical locationusing any device (e.g., including devices they own or anyshared/public/loaner device).

Work items to be edited or viewed can be accessed from the workspace inaccordance with the platform and/or application independent mechanismsdisclosed herein. Users can also be notified of access-, edit-,modification-, and/or upload-related actions performed on work items byother users or any other types of activities detected in the work space.For example, if a user modifies a document, associated collaborators canbe notified of the modification in real time, or near real time, or notin real time. The notifications can be sent through any of all of thedevices associated with a given user, in various formats including, oneor more of, email, SMS, or via a pop-up window in a user interface whichthe user uses to access the collaboration platform. In the event ofmultiple notifications, each notification can be depicted preferentially(e.g., ordering in the user interface) based on user preferences and/orrelevance to the user (e.g., implicit or explicit).

For example, a notification of download-, access-, read-, write-, edit-,or upload-related activities can be presented in a feed stream amongother notifications through a user interface on the user device 102according to relevancy to the user determined based on current or recentactivity of the user in the web-based collaboration environment.

FIG. 3 is a diagram of a system having a host server, origin server orcontent server capable of using feedback information to select an upload(or download) pathway from a number of alternative pathways. As anoption, one or more instances of diagram or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the diagram or any aspect thereofmay be implemented in any desired environment. This figure shows anexample diagram of a system having a host server, origin server orcontent server 300 (e.g., the host farm 101 in FIG. 1A1) capable ofusing feedback information to select an upload (or download) route orpathway from a number of alternative pathways for a client or userdevice 102 (e.g., user device 102 in FIG. 1A1) to optimize or enhanceupload (or download) performance between the user device and the hostserver 300.

Time to upload/download a file to/from a remote host (e.g., host server300) depends on various factors. Usually, the time is limited by auser's upload bandwidth. For example, if the user 375 pays for Internetservice that advertises 2 MB/s upload speed, then the user might expecta 20 MB file to upload in 10 seconds. However, the user can usually onlyapproach that speed if he or she is connected to a remote host that isnearby, since factors such as network congestion, the “TCP window,” andlatency to the remote host usually reduces the upload speed performance.To approach the user's theoretical maximum upload speed, the connectionbetween a client device (e.g., client device 302) and the remote host(e.g., the host server 300) usually needs to have low latency. Latencyis the time it takes for packets to make a “round trip” between theclient device (e.g., client device 302) and the remote host (e.g., thehost server 300).

Users who are further away from the remote host upload slower because ofthe way TCP (Transport Control Protocol) attempts to guess the bandwidthof a connection. When the user starts to send data, the TCP stack on theclient device sends a few packets at first, followed by more packets asit hears back from the remote host that data is arriving successfully.The success messages from the remote host are called “acknowledgements”or “ACKs”. TCP does not speed up at all until it receives ACKs. As aresult, when it takes a long time to get the ACKs, TCP spends a longertime making the client device wait, rather than sending data. Thus TCP,inherently, under-utilizes one's Internet connection early in theconnection. Furthermore, TCP backs off fast when it detects congestion,and then speeds up slowly again limited by the round trip time orlatency. As packets traverse more systems, TCP is likely to have toretransmit the packets (because of packets being lost or dropped), whichcan also slow down the connection.

The disclosed technology, in an embodiment, distributes edge nodes oredge servers to regions around the world to reduce the latency in roundtrips that the TCP protocol causes users to experience when they areuploading or downloading files from remote hosts such as the host server300. The distributed edge nodes can use two individual connections. Thedistributed edge node can be close to the end user such that the latencyof the connection is small. After the low latency connection between theend user and the edge node terminates, a second connection between theedge node and the backend can be created. This significantly reduces thelatency since the client device spends less time waiting on the TCPwindow being saturated. In other embodiments, the disclosed technologyincludes virtual machines on Amazon EC2, Microsoft Azure and other cloudinfrastructures that can effectively act as another option foraccelerating uploads and/or downloads.

As illustrated in FIG. 3, a request from the client device 302associated with the user 375 to upload (or download) files can be routedto an edge node that is in geographic proximity to the client device 302to obtain latency reduction and improvement in the user experience. Datauploaded to the distributed edge nodes 305 still have to be sent to thehost server 300. However, the connection from the edge nodes 305 to thehost server 300 can be a good connection, or at least can be aconnection that is free from bandwidth throttling or other limitations.

In addition to the direct upload route 312 whereby the user 375 canupload files directly to the host server 300 and upload via thedistributed edge nodes 305, the disclosed technology, in an embodiment,provides the user 375 an additional route 310 to upload/download filesto/from the host server 300. The additional route 310 can be one or morecontent delivery networks (CDNs) such as the CDNs provided by commercialCDNs (e.g., Akamai Technologies, Limelight Networks), Telco CDNs (e.g.,AT&T Inc., Verizon), and the like. Thus, the user 375 can upload filesto the host server 300 directly or via an edge node that isgeographically close to the user or a geographically close to thethird-party content delivery network 310. In an embodiment, the clientdevice 302 and/or the host server 300 can make a determination regardingthe optimal path available and then use the optimal path to upload ordownload files to or from the host server 300. The optimal path can be apath that optimizes certain attributes associated with upload/downloadfor a given set of client device characteristics. For example, theoptimal path can include the fastest path when speed is the factor beingoptimized. Similarly, the optimal path can include the most reliablepath when reliability is the factor being optimized. In someembodiments, more than one factor can be optimized to determine theoptimal path.

In an embodiment, the client device 302 can determine the optimal path(e.g., the fastest path from the options available (e.g., path 305, 312and 310) independent of the host server 300 based on upload feedback 314b. For example, the client device 302 can perform an upload speed test(e.g., via an upload UI or in the background) to each of the servers305, 310 and 300. The client device 302 can then compare the results ofthe upload speed test to determine and select the fastest upload path.The client device, in an embodiment, further caches the results so thatthe upload speed test need not be repeated before each upload. However,since a path that is determined to be fast at a given time can changeand not remain fast at a later time, the cache can be managed orinvalidated periodically so that the selection of the upload path isbased on results that are valid. This client-side feedback loop allowsoptimization or enhancement of the upload/download performance.

Selection of the optimal (e.g., fastest) path can be based on uploadfeedback 314 a from the host server 300. The determination of thefastest path does not interfere with the user experience and adapts todynamic conditions of the connection (e.g., Internet), but still allowsuploads to flow through the fastest path from the options available. Thehost server 300 can dynamically route uploaded files along the fastestpaths, taking advantage of the edge nodes 305, CDNs 310 and the uploadroute 312 direct to the host server 300, collect upload speed data fromevery upload location, and use that data to suggest a path for futureuploads. Thus, the server-side feedback loop allows optimization orenhancement of the upload/download performance.

FIG. 4 is a diagram 400 of a feedback loop for optimizing or enhancingupload (or download) performance. As an option, one or more instances ofdiagram 400 or any aspect thereof may be implemented in the context ofthe architecture and functionality of the embodiments described herein.Also, the diagram 400 or any aspect thereof may be implemented in anydesired environment.

In an embodiment, the feedback loop is a mechanism that collectsupload/download speed data from every upload/download location or server(e.g., the edge nodes, the CDNs, the host servers, etc.) and uses thecollected data to suggest a pathway for future uploads/downloads. Asdescribed with reference to FIG. 3, the feedback loop can be aclient-side feedback loop or a server-side feedback loop. Theclient-side feedback loop and the server-side feedback loop can each beimplemented on its own or in conjunction with each other in variousembodiments of the disclosed technology.

The feedback loop recognizes that data transfer speeds can be highlyvariable and sensitive to the environment and the context. Factors suchas the Internet Service Provider (ISP) that a user device is connectedto, the operating system of the user device and the exact browser (forbrowser-based uploads) can influence the speed of the upload/download.Such internal factors, routes, latency and packet loss characteristicsbetween the user device and the upload server can affect the uploadspeed, making the task of predicting which route might be the fastestdifficult.

The feedback loop clocks the speed of real uploads through everypathway, aggregates the upload information based on a combination offactors and when a user is about to upload, based on a given combinationof factors, it chooses a pathway with a probability such that the fasterpathways are reused much more often, and the slow ones are retried justfrequently enough to not penalize them. This method allows for selectionof a faster pathway much more often than a slower path. In other words,the bulk of the traffic is sent through faster pathways but the slowerpathways can be retried occasionally to validate that their performancehas not changed. This retry mechanism allows the system to reassess thepathways and avoid making a poor choice if other pathways improve. Thesame is true if the speed recorded for a slow pathway was anomalous.

As illustrated in FIG. 4, in an embodiment, uploads 405 from clientdevices (e.g., client device 302 in FIG. 3) via various pathways arereceived at the backend 410 (e.g., the host server 300 in FIG. 3). As anupload is received at the backend 410, the timestamps when the first andlast bytes of the upload reach the server-side are recorded. The hostserver then uses the timestamp information to compute or estimate theupload speed 415 at which the upload occurred. The upload speed 415 andother signals, indications, attributes or characteristics about theuser, the upload and/or the client device from where the uploadoriginated are recorded in an analytics datastore 430. The loggedinformation is used to perform analysis after the fact about signalsthat can be used in the future to make decisions in real time.

The feedback loop includes an aggregator that aggregates upload speedsfor each pathway using a combination of factors (or a set ofcharacteristics or attributes). The aggregate upload speeds are storedin a key/value store 420, where the combination of factors is the key oridentifier to a record that holds as value an array of pathways or hostsand upload speeds associated with the pathways. The key/value store 420provides fast access to recent information about the speed of uploadsvia each pathway. Further, since the upload speeds are aggregated usinga combination of factors, users that are likely to perform similarly aregrouped together. This allows one user's experience about an upload toinfluence the upload choices of other users in the same grouping. Forexample, when two users at the same location, who have the sameoperating system and browser combination, are uploading files to thesame host server (e.g., using the same Wi-Fi connection), their uploadswill influence the aggregator. This means that when the next user havingthe same location, browser and operating system characteristics is aboutto upload a file, the information from the two users can be used todetermine where the next user should upload to. While doing so, thefeedback loop remains entirely decoupled from individual users. Forexample, when a user uploads from his or her home connection, the useris using a different set of aggregates compared to when the user isuploading from their work connection, and upload data from the “home”connection is aggregated in a bucket separate from the “work”connection.

Tables 1 and 2 below provide examples of two aggregators. Table 1 showsa first aggregator after 21 files have been uploaded from a location inVirginia using client devices having a specific combination of browserand operating system platform. As depicted in Table 1, after an initialobservation period during which pathways were tried using round robinselection, most of the uploads were through the “DC Edge Node” which hassignificantly higher upload speed than the CDN or the direct to hostserver.

TABLE 1 Aggregator with optimization using real-time feedback loopUpload Route Aggregate Upload Speed Upload Count DC Edge Node 25,816kbps (3,227 KB/s) 16 CDN 9,839 kbps (1,230 KB/s) 3 Direct to Host Server1,222 kbps (153 KB/s) 2

Table 2 below shows a second aggregator after 5 files have been uploadedfrom a location in Connecticut using a client device. As depicted inTable 2, the upload speeds across each of the pathways are comparable,indicating that the Internet connection was throttled, and thus there isno optimization to be provided. The upload count indicates that uploadroute is selected using round robin.

TABLE 2 Aggregator without Optimization Upload Route Aggregate UploadSpeed Upload Count CDN 2,082 kbps (260 KB/s) 1 Direct to Host Server2,070 kbps (259 KB/s) 2 DC Edge Node 2,049 kbps (256 KB/s) 2

The key/value store 420 can be implemented as a non-persistent datastoresuch as memcache or other caching system. The key/value store 420 canstore, as an array, information such as an aggregate upload/downloadspeed for each pathway and a frequency with which each pathway was usedfor upload/download in association with a key that can comprise a set ofclient device/user characteristics such as an IP address (e.g., firstthree or any number of octets), an operating system version, a browserversion, a time period, and/or a combination of any of the above. Forexample, all upload data for a given day from client devices having IPaddresses 24.24.24.XX, Windows 7 and Internet Explorer v. 8.0 areaggregated and stored under the key (24.24.24.XX, Windows 7, InternetExplorer v. 8.0). The value stored under each key is updated as newupload/download data becomes available.

In a web application, for example, a web client (or mobile client) cansend a request to get an upload host suggestion from the feedback loopat upload time. When the upload request is detected, the feedback loopaccesses the key/value store 420 to review key/values for a given timeperiod (e.g., 7 days), and selects a pathway that has the maximum orhigher normalized sum of speeds over the given time period.

The key/value store 420 can be configured to have a certain size,depending on the size of each entry in the key/value store, the numberof possible keys, number of operating system versions, number of browserversions and number of days' worth of data. For example, if each entryin the key/value store 420 is estimated to have a size of 250 bytes, thenumber of keys possible is 224 (i.e., first three octets of the IPaddress), and if three OS versions and six browser versions are beingrecorded for 7 days, the theoretical maximum size of the key/value store420 can be determined to be:

2²⁴×3×6×250B×7≈492 GB

In addition to recent data being stored in the memcache key/value store420, the system also includes the persistent analytics cluster oranalytics datastore 430 that stores all analytics data. In oneembodiment, data stored in the analytics cluster can include, but is notlimited to: IP address (e.g., unsigned int), upload/download session id(or connection diagnostics ID), user ID, enterprise ID, upload/downloadsize (bytes), upload/download duration (milliseconds), upload/downloadspeed (e.g., in kbps), upload/download start timestamp (epoch seconds),upload/download proxy (maps to class constants), platform and/or majorversion, browser and/or major version, upload/download type (e.g.,HTML5, AJAX, connection diagnostic test, class constants), country,region, and/or the like. When the information in the key/value store 420expires, or when a total memcached flush occurs (unintentional orotherwise), the data stored in the analytics datastore 430 can be usedto repopulate the entries in the key/value store 420.

FIG. 5A presents a diagram 5A00 of a feedback loop for optimizing orenhancing upload (or download) performance. As an option, one or moreinstances of diagram 5A00 or any aspect thereof may be implemented inthe context of the architecture and functionality of the embodimentsdescribed herein. Also, the diagram 5A00 or any aspect thereof may beimplemented in any desired environment. This figure illustrates a blockdiagram depicting example components of a host server 500 capable ofusing feedback information to select a pathway from a number ofalternative pathways for a client device to upload (or download) data tooptimize or enhance upload (or download) performance.

In an embodiment, the host server 500 of the web-based or onlinecollaboration environment can generally be a cloud-based service (e.g.,the host farm 101 in FIG. 1A1). In an alternate embodiment, the hostserver 500 can be any host server or content server from where data canbe downloaded or to where data can be uploaded. The host server 500 caninclude, for example, a network interface 505, an upload requestprocessor 506 having a drag-drop manager 508, an upload session tracker510 and an upload start request data extractor 512, an upload engine 514having a multi-file upload manager 516 and a folder upload manager 518,a notification engine 520 having a feed stream updater (e.g., see theshown feed stream updator 522) and a recipient selector 524, a userinterface module 526 having a navigation manager 528 and an uploadedcontent access module 530, a feedback loop optimizer 550, and an uploadanalytics engine 532 having an upload data analysis module 534 and anadmin user interface 536. One or more components of the host farm 101can be in communication with or coupled to a persistent analyticscluster or datastore, and one or more memcached key/value stores can bein communication with or coupled to memcached servers.

The network interface 505 can be a networking module that enables thehost server 500 to mediate data in a network with an entity that isexternal to the host server 500, through any known and/or convenientcommunications protocol supported by the host and the external entity.The network interface 505 can include one or more of a network adaptorcard, a wireless network interface card (e.g., SMS interface, Wi-Fiinterface, interfaces for various generations of mobile communicationstandards including but not limited to 1G, 2G, 3G, 3.5G, 4G, LTE),Bluetooth, a router, an access point, a wireless router, a switch, amultilayer switch, a protocol converter, a gateway, a bridge, a bridgerouter, a hub, a digital media receiver, and/or a repeater. The externalentity can be any device capable of communicating with the host server500, and can include client user devices 102, sync server 120,notification server 150, distributed edge nodes 104, content deliverynetworks 110, and the like illustrated in FIG. 1.

An embodiment of the host server 500 includes the upload requestprocessor 506 which can receive, detect, process, identify, parse,extract, translate, and/or determine an upload start request ornotification and/or an actual upload from a client device. The uploadrequest can be submitted by a user (e.g., through a user interface of aweb-based or mobile application) to upload one or multiple items. Theupload start request (or initial request for upload pathway) can be arun mode request that reaches the host server prior to the actualupload. The upload start request can inform the host server of theupload and provide information such as the IP address, browser version,platform version of the client device, or the like. The upload startrequest can allow the host server to keep track of failures or otherproblems with the upload. The upload start request can also act as arequest to the host server to provide an optimal upload pathway for theclient device to upload files. The host server can determine the optimalupload pathway via the feedback loop optimizer 550 and provide theoptimal upload pathway to the client device. The feedback loop optimizer550 is described in detail in FIG. 5B. The client device can then uploadthe selected file or files to the optimal upload pathway. In someembodiments, instead of an optimal upload pathway, the host server canprovide the client device a weighting or other information using whichthe client device can itself select an optimal upload pathway using analgorithm (e.g., random weighted selection method).

When the upload start request is detected by the upload requestprocessor 506 (e.g., using a Perl script), information such as the IPaddress, browser and platform information (e.g., name, version), and thelike in the upload start request (e.g., HTTP or HTTPS request) can beparsed and extracted by the upload start request data extractor 512, anda new entry for the upload can be created in the analytics cluster 502and/or the memcached servers to store the upload data. In an embodiment,a session ID can be created by the upload session tracker 510. Thesession ID can be provided to the client device and may be stored alongwith the IP address in the analytics cluster 502 and/or the memcachedservers. The session ID allows the host server 500 to associate anupload to the corresponding IP address. This can be advantageous incases where an upload pathway via which the upload is routed wipes outthe IP address of the client device. When a session ID is included inthe upload, even if the IP address is wiped out, the host server 500 canuse the session ID to determine the IP address associated with theupload.

The user can identify the files, content, or items to be uploaded to thehost server 500 one-by-one and queue up multiple items (e.g., includingbut not limited to files, folders, documents, images, audio) to beuploaded in a single request. The user can also select all of the itemsto be uploaded in a single action (e.g., via highlighting or otherwiseselecting of icons corresponding to each of the items). In oneembodiment, the upload request is generated via a drag-and-drop actionof the multiple work items to be uploaded to the host server into aportion of the user interface. Drag-and-drop activated uploaded requestscan be detected, handled, received, processed, and/or otherwise managedby the drag-drop manager 508.

In an embodiment, the upload request is generated via a drag-and-dropaction of a single folder which includes the multiple work items to beuploaded to the host server 500. For example, the upload request can begenerated when a folder having the multiple items on a client devicethat is to be uploaded is identified through the user interface. In someinstances, the folder can include additional folders in a folderhierarchy of multiple items. In some instances, the user can generate anupload request by activating the upload feature in a tab on the userinterface and initiate uploading by selecting (e.g., clicking on orotherwise activating) the button/tab. Once selected, another userinterface or a pop-up window may appear allowing the user to navigatethrough files or folders to select the items to be uploaded.

Once upload requests have been detected and processed, the upload engine514 can upload the requested item or multiple requested items. Theupload engine 514, in an embodiment, uploads a single item or multipleitems (e.g., sequentially or simultaneously) to the host server 500 viathe server-selected (or client-selected) upload pathway. A multiple itemupload may be initiated via a single-step or multi-step user request. Amulti-file upload request can be handled, processed, and executed, forexample, through the multi-file upload manager 516.

In one embodiment, the multi-file upload manager 516 receives anidentification of each of the multiple files to be uploaded (e.g., fromthe upload request processor 506) and sequentially prepares eachindividual file for uploading and uploads each file independently. Forexample, the multi-file upload manager 516 can compress one of themultiple files individually, upload it to the host server 500 anddecompress the file when uploaded and proceed to perform the same stepswith the next file. Preprocessing a file can include, for example,analyzing the file size and type to determine if it is acceptable/validand/or to identify how best to compress the file. Post-processing caninclude, for example, performing one or more of, decompressing the file,validating the file size and name, checking permissions, potentiallyscanning for malicious software, and/or moving to permanent storage. Thestep of moving to storage can further include, one or more of, addingthe file metadata to the database, creating thumbnails, creatingpreviews, indexing for search, encrypting the file, and/or storing inmultiple locations for redundancy. Note that the above processes canoccur in any order or synchronously in any combination with one another.The process continues until all items in the request have been uploadedto the host server 500. The upload may automatically progress from onefile when completed to the next one in sequence when the user initiatesa multi-file upload request.

In one embodiment, the upload engine 514 uploads multiple items in afolder hierarchy based on a single request to upload a folder which hasa hierarchy of folders inside, for example, via the folder uploadmanager 518. In one embodiment, the folder upload manager compresses themultiple items in the folder hierarchy in a single process into a singleitem and uploads the single item in a single upload process (rather thanone by one) to the host server 500. After the merged file of multipleitems has been uploaded, the folder upload manager 518 can decompressand subsequently parse the single upload of the single item into theoriginal individual files that were stored as multiple items in thefolders in the hierarchy. By merging multiple files into one andperforming a single compression, and decompression step, the uploadingprocess can be expedited since the overhead in time to compress anddecompress multiple files is mostly eliminated. Some additional benefitsof bulk uploading allow the following overhead to be partially or whollyeliminated: repeatedly creating TCP connections for each upload,repeatedly checking the same permissions and storage quotas whenprocessing the files on the server.

One embodiment of the host server 500 includes the user experience/userinterface module 526, which preserves or enhances user experiencebefore, during, or after an upload request. For example, the userexperience/user interface module 526 (UE/UI module) can allow the userto engage in other activities in the collaboration platform while anupload is in progress so as to prevent the user from having to wait forthe completion to work in the platform.

In one embodiment, during the upload of a single file (beforecompletion), the user can generally navigate away from the userinterface through which the upload request was submitted, for example,via the navigation manager 528 in the user experience/user interfacemodule 526. In other words, while a file or item upload is in progress,the user can navigate to other pages to perform other actions orinitiate additional actions on the current page without interrupting(stopping or pausing) the in-progress upload.

Similarly, when a multi-file or multi-item upload request is inprogress, the user can also navigate away from the user interface whichthe upload request was submitted prior to completion of the uploading ofeach of the multiple items to the host server 500 via an acceleratornode. Navigation between pages during an upload of multiple files canalso be managed by the navigation manager 528. For example, the uploadof the multiple items can continue to proceed and is not interrupted ifthe user accesses a link on the user interface causing another userinterface to launch in a browser. To enable bulk uploading, a newbrowser window is opened so it operates independently of usernavigation. In addition, the web application for uploading and access ofthe collaboration environment is “pageless”, meaning it can be updatedasynchronously without a browser page refresh. This allows navigationand to start new uploads in other folders, which can be added to theupload queue.

In addition, during a multi-file upload, an item of the multiple itemsthat has been uploaded to the host server 500 available for accessthrough the user interface, even when some of the multiple items havenot yet been uploaded to the host server, via the upload content accessmodule 530, for example. Thus, during an active upload, individual fileswhich have completed uploading can be accessed or interacted with by theuser in the collaborative environment without having to wait for thefull upload to complete.

In some instances, the item which has been uploaded to the host serveris manipulatable by the user through the user interface, without a needfor browser refresh. This enhances the user experience by allowing theuser to work on the file or otherwise interact with it once it has beenuploaded without waiting for other files to finish uploading. Forexample, the user can view, edit, preview, or comment on the item thathas been uploaded, prior to completion of uploading all of the multipleitems in an upload request. In one embodiment, buffer space in memoryfor storage of the individual work items are created in response to theupload request such that when individual items have been uploaded, theycan be moved into the created buffer space, and subsequently permanentstorage. When the file is in permanent storage, the user can then accessand work on the individual item, while others are still being uploaded.In one embodiment, metadata for the file can be created before it isfully uploaded or processed, allowing faster user interaction. However,to actually interact with the file content (full content search,download or preview) the file generally needs to be processed as usualand be stored in permanent storage.

One embodiment of the host server 500 includes a notification engine520. The notification engine 520, can for example, update a feed streamto include an updated feed indicating that an item or multiple itemshave been uploaded, for example, via the feed stream updator 522. Theusers that are notified can be selected, for example, by the recipientselector 524, and can include collaborators or the user, or other usersmeeting a criterion. In some instances, the feed stream is updated inreal time or near real time relative to when the upload of the itemcompleted. For real-time updating, the notification engine 520 can useanother server, or another engine in the same server which provides pushfunctionality.

The notification engine 520 can generally inform or notify users, whichcan be collaborators of the user who performed the activity in the workspace via one or more of many mechanisms, including but not limited to,email, SMS, voice-message, text-based message, RSS, feed, and the like.

In one embodiment, the notification is depicted through a web-browserused by the other user to access the web-based collaborationenvironment, for access in real time or near real time to when theactivity was performed by the user. When notifying a user in real timethrough a web-browser, the notification engine 520 can use apush-enabled service to ensure real time notification. In oneembodiment, the notification is sent by a component or another serverwhich implements push technology. The push-enabled service can beimplemented via long poll or HTTP streaming, for example, by a devicewhich may be internal to or external to the host server 500. Inaddition, the host server 500 could use other push servers includingthird party push servers to implement push technology including but notlimited to mobile platform push systems and services (e.g., via smartphones or tablets or other portable devices such as iPhone, Androidphones, Blackberry, iPad, Galaxy or other tablets)

The upload analytics engine 532 can parse, process, and/or analyze datarelating to uploads/downloads stored in the analytics cluster 502. Theupload data analysis module 534 can perform analytics operations,including statistical analysis, on the stored data. For example, theupload data analysis module 534 can analyze upload information todetermine the effectiveness of the memcached speed aggregator, determineadjustments that can be made (e.g., change the time over which data isaggregated, determine if any characteristics are more/less relevant, andthe like) and make the adjustments. By way of another example, theupload data analysis module 534 can determine percentage of users (e.g.,total users, enterprise account users, free account users) that map toedge nodes, CDNs and/or the host server. Similarly, the upload dataanalysis module 534 can further determine for each region mapped to anedge server, the percentage of users who are benefitting the edge serverat any given time, percentage of users who are instead experiencingfaster uploads via a CDN server or directly to the host server. Theseexamples are merely illustrative, and additional analyses on theupload/download data may be performed.

FIG. 5B presents a block diagram showing instances of selectedcomponents 5B00 of a feedback loop optimizer. As an option, one or moreinstances of selected components 5B00 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the selected components 5B00 or anyaspect thereof may be implemented in any desired environment. Thisfigure illustrates a block diagram depicting example components of thefeedback loop optimizer 550 of FIG. 5A. The feedback optimizer 550 mayinclude a feedback aggregator engine 552 having an aggregation profilecreator 554, an aggregation profile updator 556, a feedback aggregatorfilter module 558 and/or an aggregate data invalidator module 560. Thefeedback optimizer 550 may also include a speed calculator 562, anoptimization parameter configuration module 564, a pathway selectionengine 566 having a probability-based pathway selection engine 568, aweighted random choice based pathway selection engine 570, a mode changetrigger module 571, and/or a data logger 572. More or less componentsmay be present in the feedback loop optimizer 550 in some embodiments.One or more of the components of the feedback loop optimizer 550 may beconsolidated into a single component, in some embodiments.

The data logger 572 can log data relating to uploads/downloads via eachpathway. The data logger 572 can further recognize and extractinformation relating to uploads/downloads for recording. Theupload/download data logged by the data logger 572 can be stored in theanalytics cluster 502. The data logger 572 can store various fields ofupload/download data such as, but not limited to: the IP address,upload/download session ID, user id (for personal account), enterpriseid (for enterprise account), upload/download size, upload/downloadduration, upload/download speed, upload/download start timestamp,upload/download host or pathway, platform, platform major version,browser, browser major version, upload/download type (e.g., HTML5, AJAX,Connection Diagnostic Test), geographic information (e.g., country,region, zip code), and the like.

In an embodiment, the upload/download speed can be calculatedserver-side via the speed calculator 562 and stored in the analyticscluster 502. The speed calculator 562 can determine the upload/downloadspeed for each upload/download through each pathway. For example, thespeed calculator 562 can use the timestamp for the first byte and thelast byte of an upload to determine an upload duration. The speedcalculator 562 can then use the upload duration and the upload size todetermine the upload speed. In an embodiment, the speed can becalculated on the client-side (e.g., using upload/download speed test).The host server 500 can accept the results of the speed test performedclient-side and can insert the result in its datastore (e.g., analyticscluster 502 and/or the memcache 504 along with speed determinedserver-side. In some instances, the host server can set the upload typefor the speed reported by a client device to describe the method thatwas used to determine the speed (e.g., connection diagnostic test). Thefeedback loop optimizer 550 can use both types of speed data whenselecting a pathway to recommend.

The feedback aggregator engine 552 includes the memcache 504 (orkey/value store). The feedback aggregator engine 552 can aggregate andstore in the memcache 504, recent and/or relevant data relating touploads/downloads through various pathways. The feedback aggregatorengine 552 can aggregate upload/download data based on a combination offactors such that users or client devices that are likely to performsimilarly and/or have similar set of characteristics, features orattributes are grouped together. The data from the feedback aggregatorengine 552 can be processed and/or analyzed to determine optimalpathways for a given uploader for uploading/downloading files.

In an embodiment, the feedback aggregator engine 552 can aggregate orgroup upload/download data associated with uploads/downloads througheach pathway into aggregation profiles via the aggregation profilecreator 554. Each aggregation profile is defined by a combination offactors such as the IP address (e.g., first 3 octets), OS version,browser version and/or time period. For example, the feedback aggregatorengine 552 can aggregate or group together upload data for uploads fromthe same IP address (e.g., 67.123.129.XX), same browser major version(e.g., Google Chrome Version 27.0) and same OS major version (e.g.,Windows 7 Enterprise) through the same upload pathway (e.g., Chicagoedge server). A structure of an aggregation profile defined by IPaddress, OS, browser and time period information that comprisescorresponding upload information (e.g., upload speed, upload count)associated with an array of pathways is illustrated below:

Key(IP/24, OS, Browser, Day)=>array(

aggregate edge node upload speed,

number of edge node uploads,

aggregate CDN upload speed,

number of CDN uploads,

aggregate direct upload speed,

number of direct uploads,

. . .

)

In an embodiment, the aggregation profile updator 556 can update orrecalculate the aggregate upload/download speed and upload/downloadcount (i.e., number of uploads or downloads) as new uploads/downloadscorresponding to an aggregation profile are received. For example, thehost server 500 receives an upload via the Portland edge server andhaving a given combination of IP address, browser and OS (e.g.,123.43.234.XX, Chrome, MacOSx). The speed calculator 562 determines theupload speed to be 8,000 kbps. The aggregation profile updator 556 thenlooks up the aggregation profile for the given combination of factorsand updates the aggregate upload speed and the upload count for thegiven combination of IP address, browser and OS from a previous value(e.g., aggregate edge node upload speed=5375 kbps and number of edgenode uploads=10) to a new or updated value (e.g., aggregate edge nodeupload speed=5613.64 and number of edge node uploads=11). The newaggregate may be determined using a method substantially similar to themethod illustrated below:

new_aggregate=[(aggregate_speed*number_of_uploads)+new_upload_speed]/(number_of_uploads+1)new_number_of_uploads=number_of_uploads+1

In an embodiment, the feedback aggregator filter module 558 can includeone or more filters to ensure that the feedback aggregator engine 552does not accept upload/download data that are unusable. For example, afile size filter can be implemented when aggregating data that is usedfor selecting a upload/download pathway. The file size filter canrequire uploads/downloads to be over a certain size in order toinfluence the feedback loop. If the uploads/downloads are too small, theTCP window does not have the opportunity to grow sufficiently and thebenefits of the edge servers and/or CDN servers cannot be leveragedfully or at all. A threshold for determining which upload/downloadspeeds are outliers can be determined based on empirical or otheranalysis for example. Another filter that can be used is a filter thatputs a limit on the derived upload/download speed. For example, if afile is too small, the file can land in the OS buffers and can cause thehost server 500 (or the speed calculator 562) to clock the speed asunreasonably fast (e.g., above a threshold). The derived upload speedfilter can filter out or mark the data as being unusable to prevent suchdata from influencing the feedback loop. In one implementation, anupload speed threshold or size threshold for filtering out outliers canbe established and/or periodically adjusted based on analysis ofhistorical upload/download data, determination of the theoreticalmaximum upload/download speed, and the like, for example.

In an embodiment, the information (e.g., the aggregation profiles) inthe memcache 504 that is used to select pathway to upload/download datais marked with an expire time and can become invalid or evicted from thememcache 504 after a certain time, as new information associated withmore recent uploads/downloads is added to the memcache 504. Theaggregate data invalidator 560 can track a timing attribute (e.g., TTLvalue) associated with the information in the memcache and caninvalidate the information in the memcache 504 that is out of date orexpired to avoid such out of date data from being used in the feedbackloop. In an embodiment, the aggregate data invalidator 560 maintains xnumber of days (or any unit of time, such as 24 hours) of data for usein selecting a pathway for upload/download.

In an embodiment, the feedback loop optimizer 550 optimizesupload/download speed in selecting a pathway for upload/download for agiven set of characteristics (e.g., IP address, browser information andOS information). The feedback loop optimizer 550 can also optimize otherparameters, in addition to or instead of the upload speed in some otherembodiments. The optimization parameter configuration module 564 canconfigure the pathway selection engine 566 to select a pathway based onone or more parameters. For example, the pathway selection engine 566can be configured via the optimization parameter configuration module564 to provide an array of pathways and corresponding upload/downloadspeeds (e.g., “Denver Edge node: 7284, CDN: 3905, Direct: 5932”), thebest pathway by upload/download speed (e.g., “Denver Edge node”), thebest pathway by votes (e.g., “CDN”), and the like in response to arequest for an upload/download pathway.

In an embodiment, the pathway selection engine 566 receives specifictypes of requests for recommendations. For example, the pathwayselection engine 566 can receive a request for best pathway by speed(e.g., get_best_hosts_by_speed( ) with IP/24 address, OS version andbrowser version as parameters for the request). In response, the pathwayselection engine 566 provides, for the given set of characteristics, anarray of hosts and corresponding speeds (e.g., “Chicago Edge node: 7284,CDN: 3905, Direct: 5932”). Similarly, the pathway selective engine 566can receive a request for the best pathway for an upload/download (e.g.,get_best_host( ) with IP/24 address, OS version and browser version asparameters for the request). In response, the pathway selection engine566 provides the best pathway by speed for the given set ofcharacteristics (e.g., “Chicago Edge node” based on the above example).Other modes of recommendation such as recommendation by votes, uploadtype, connection type, file size, criticality of upload data, type ofaccount (e.g., personal or enterprise account), type of account (e.g.,free or paid), and/or the like may be implemented by the optimizationparameter configuration module 564 and/or the pathway selection engine566.

In an embodiment, the pathway selection engine 566 selects a pathwayusing feedback information to optimize or enhance upload and/or downloadperformance. In other embodiments, the pathway selection engine 566provides feedback information (e.g., pathway options) to a client deviceto use in selecting a pathway to optimize or enhance upload and/ordownload performance. The selection of a pathway is based on informationfrom the feedback aggregator engine 552 stored in the memcache 504. Thepathway selection engine 566 can select an upload/download pathway basedon various methods or algorithms. In an embodiment, the pathwayselection engine 566 implements a weighted random selection algorithmvia the weighted random selection-based engine 570. The weighted randomselection-based engine 570 can select a faster pathway more frequentlythan a slower pathway based on the weights associated with the pathways.As an example, the upload speed through pathway A is 2 mbps and theupload speed through pathway B is 1 mbps. The weighted randomselection-based engine 570 can select pathway A with a probability of(2/(2+1)) or ⅔ and pathway B with a probability of (142+1)) or ⅓. Thusthe faster pathway is selected more frequently than the slower pathway,but once in a while the slower pathway is also tried to check if thespeed has improved. This effect can be enhanced by increasing theexponent for the weights. For example, when the speed of pathways A andB are each squared, the weighted random selected-based engine 570 canselect pathway A (2²) or 4 out of (2²+1²) or 5 times, and pathway B,(1²) out of (2²+1²) or 5 times. Similarly, when using a cubed factor,the speed of pathway A and pathway B can be cubed and added together todetermine the total weight of (2³+1³) or 9. The weighted randomselection-based engine 570 can have (2³) or 8 out of 9 files be uploadthrough pathway A, and (1³) or 1 out of 9 files be uploaded throughpathway B. This method keeps most uploads pointed to the fastest orfaster known pathways, while collecting data from other pathways. Whennetwork shifts, the weighted-random selection-based engine 570 candetect any change in upload speeds associated with the pathways andadjust its recommendation accordingly.

In an embodiment, the size of the file to be uploaded or downloaded canbe factored in when computing the weights such that the higher weightsare associated with increasing file sizes. The weighting can beconfigured and tweaked such that files over a certain size wouldapproach utilization of the currently fastest known host at near 100% ofthe time. In some instances, a file size threshold can be set such thatfiles reaching or exceeding the threshold size get uploaded ordownloaded via the fastest known host. In general, the threshold size isconfigurable based on the client, customer, subscription or othercriteria.

In an embodiment, the pathway selection engine 566 can select a pathwaybased on equal probability via the equal probability based pathwayselection engine 568. In other words, this method accords each of thepathways equal weight, such that each pathway has an equal probabilityof being selected for an upload or download.

In an embodiment, a mode change trigger detector 571 can switch betweenthe equal probability based pathway selection engine 568 and theweighted random selection-based engine 570 based on one or more criteriasuch as availability of data. For example, when a given set ofcharacteristics (i.e., IP/platform/browser) lacks a minimum number ofdata points (e.g., upload speeds for at least two hosts), the pathwayselection engine 566 does not have enough information to determine whichhost is faster or better pathway. The pathway selection engine 566 thenuses a round-robin method to select a pathway, which may or may not bethe optimal pathway. The mode change trigger detector 571 can detectwhen enough information or minimum number of data points has accumulated(e.g., in the memcache) and can shift the pathway selection methodologyfrom the equal probability based selection to weighted random selection.In another embodiment, the mode change trigger detector 571 can includea throttling detector (not shown) that can detect whether a connectionbeing throttled or has a bandwidth cap or other limitations that canprevent optimization of the upload/download performance. If thethrottling detector detects that a connection is throttled (e.g., basedon speed test results reported by the client device), the mode changetrigger detector 571 can trigger the equal probability-based pathwayselection engine 568.

In an embodiment, the feedback loop optimizer 550 includes the pathwayselection configuration module 574. The configuration module 574 caninclude several options for configuring or customizing the pathwayselection engine 566. For example, the configuration module 574 can beused to set the minimum number of data points for a given set ofcharacteristics needed to shift from equal probability-based pathwayselection method (via engine 568) to weighted random choice-basedselection (via engine 570). The configuration module 574 can also beused to define an analysis window. An analysis window defines the numberof days (e.g., 7 days or any other time period such as 24 hours) of datathat can be considered when the pathway selection engine 566 isrecommending a best host for a client matching a given set ofcharacteristics. The data corresponding to the analysis window can thenremain in the memcache, making access to the data much faster. Theconfiguration module 574 can also be used to configure the exponentsetting for determining the weights for nodes. Any exponent including asquare, a cube, and so on may be used by the weighted randomchoice-based selection engine 570 in determining an optimal host forupload/download. The configuration module 574 may also be used toconfigure a recommendation time to live (TTL) value. The recommendationTTL value can specify how long the recommendations or selections fromthe pathway selection engine 566 can be cached and reused by clients.For example, an array of hosts and corresponding weights for the arrayof hosts provided as recommendation by the pathway selection engine 566can have a TTL value of 24 hours. The client device can use the cachedweights during the 24-hour period to determine an upload host each timea file is to be uploaded, obviating the need to request the host serverto provide a recommendation before each upload. Thus caching ofrecommendations or selections can help avoid expensive recalculations,or the need to request a pathway or weights for pathways each time afile is to be uploaded/downloaded.

In an embodiment, the pathway selection configuration module 574includes an opt out (or opt in) module (not shown). Using the opt outmodule, certain users can opt out of certain classifications of host oraccelerator nodes. For example, certain users can opt out of Akamai CDN.Thus, for those users, the Akamai CDN is removed from the eligible hostslist prior to running the pathway selection engine 566. In oneimplementation, opt out can be a preference that is specified by theuser (e.g., via a user request when requesting for a hostrecommendation). In another implementation, the opt out may be systemconfigured. For example, users who pay for the services provided by thehost server can be recommended a host from a first list of hosts thatincludes direct pathway, CDN servers and edge servers. Similarly, userswho do not pay for the services provided by the host server can berecommended a host from a second list of hosts (e.g., a shorter ordifferent list of hosts). For example, “free users” can be recommended ahost from a list that includes direct pathway and edge servers. In someembodiments, opt in/opt out configuration may be based on otherattributes such as type of user account (e.g., enterprise or personal),geography, membership duration (e.g., users for 6 months), promotions,and the like.

FIG. 6A and FIG. 6B are block diagrams, presented as diagrams 6A00 anddiagrams 6B00, respectively. FIG. 6A is a block diagram depictingexample components of a user device that provides some of the feedbackinformation that is used by a host server to select a pathway toupload/download files to optimize or enhance upload and/or downloadperformance. FIG. 6B is a block diagram showing example components of aclient-side feedback loop optimizer used for dynamically shaping networktraffic when using full-duplex websocket communications. FIG. 6A andFIG. 6B depict example components of a user device that provides some ofthe feedback information that is used by a host server to select apathway to upload/download files to optimize or enhance upload and/ordownload performance. As an option, one or more instances of blockdiagrams or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the block diagrams or any aspect thereof may be implemented in anydesired environment.

FIG. 6A illustrates a block diagram depicting example components of auser device 602 (e.g., user device 102 in FIG. 1A1) that provides someof the feedback information that is used by a host server (e.g., hostserver 500 in FIG. 5A) to select a pathway to upload/download files tooptimize or enhance upload and/or download performance. FIG. 6Billustrates a block diagram depicting example components of aclient-side feedback loop optimizer of FIG. 6A.

In an embodiment, the user device 402 includes components such as anetwork interface 604, a browser engine 606, a cache manager 610, a userrequest manager 612, an upload manager 614 having an upload startrequest module 616 and an upload module 618 and/or a client-sidefeedback loop optimizer 622. Additional or less components may beincluded in the user device in other embodiments. It should be notedthat the user device 602 can process upload requests, download requestsand/or both upload and download requests via the illustrated componentsor separate components that are similar to the illustrated components.

The network interface 604 may be a networking module that enables theuser device 602 to mediate data in a network with an entity that isexternal to the host server, through any known and/or convenientcommunications protocol supported by the host server and the externalentity. The network interface 604 can include one or more of a networkadaptor card, a wireless network interface card (e.g., SMS interface,Wi-Fi interface, interfaces for various generations of mobilecommunication standards including but not limited to 1G, 2G, 3G, 3.5G,4G, LTE), Bluetooth, a router, an access point, a wireless router, aswitch, a multilayer switch, a protocol converter, a gateway, a bridge,a bridge router, a hub, a digital media receiver, a repeater, and/or thelike.

As used herein, a “module,” a “manager,” a “optimizer,” an “interface,”an “engine” or a “system” includes a general purpose, dedicated orshared processor and, typically, firmware or software modules that areexecuted by the processor. Depending upon implementation-specific orother considerations, the module, manager, receiver, interface,selector, engine or system can be centralized or its functionalitydistributed. The module, manager, receiver, interface, selector, engineor system can include general or special purpose hardware, firmware, orsoftware embodied in a computer-readable (storage) medium for executionby the processor. As used herein, a computer-readable medium orcomputer-readable storage medium is intended to include all mediums thatare statutory (e.g., in the United States, under 35 U.S.C. 101), and tospecifically exclude all mediums that are non-statutory in nature to theextent that the exclusion is necessary for a claim that includes thecomputer-readable (storage) medium to be valid. Known statutorycomputer-readable mediums include hardware (e.g., registers, randomaccess memory (RAM), non-volatile (NV) storage, to name a few), but mayor may not be limited to hardware.

The user request manager 612 can detect, receive, manage, process,identify any request to upload one or more files/folders to the hostserver (or another remote device or server) and/or download remotelyhosted or stored files/folders at the user device 602. Detection of theupload/download request can cause other components to request andreceive identification of a suitable host through which the files can beuploaded from the user device 602 to the host server or downloaded fromthe host server to the user device 602.

In an embodiment, the upload manager 614 can receive informationregarding a pathway or an array of pathways from the host server, andupload one or more files to the identified pathway. The upload startrequest module 616 can send a request or a message to notify the hostserver that an upload is about to start. In an embodiment, the uploadstart request can also serve the purpose of notifying the host serverregarding the upload about to occur so that the host server can trackthe upload for any failures. In an embodiment, the upload start requestmodule 616 may make an explicit request for a host recommendation sothat the user device can upload to the recommended host and obtainbetter upload performance and user experience. The upload start requestmodule 616 may generate and send the request in the form of an HTTP, APIor other request. The request may include information about the userdevice 602 and/or the upload itself. For example, the upload startrequest can include an IP address of the user device, OS name andversion and/or the browser name and version. In some implementations,the upload start request can further include number of files selectedfor upload, size of files (total size and/or size per file) selected forupload, user device make and/or model, user details (e.g.,user/enterprise ID), and the like. The host server can receive theupload start request and in response, can recommend a host to upload toor provide information that the user device can use to select a host toupload to. The feedback information from the host server can be receivedby the feedback information receiver 624 of the client-side feedbackloop optimizer 622 illustrated in FIG. 6B. The feedback information caninclude an identification of the pathway to which the user device canupload to and/or weights (or ratios) for an array of pathways to use inselecting a pathway to upload to. In an embodiment, the feedbackinformation from the host server can be associated with a timingattribute (e.g., TTL value) that indicates how long the information isconsidered valid or usable. For example, the feedback information canremain valid or usable for up to 2 hours, 12 hours, 24 hours, 2 days, 7days, etc. The timing attribute for the feedback information can beconfigured by the host server and/or the user device. The feedbackinformation along with the timing attribute can be stored in a localcache 608 by the cache manager 610. Caching the feedback informationfrom the host server means that the user device does not have to requestand/or wait for a recommendation before each upload. In an embodiment,whenever there is a change in network conditions (e.g., upload speed athost B has changed such that host B is now faster than host A), updatedfeedback information can be pushed by the host server to the user device602. The updated feedback information can also be received by thefeedback information receiver 624.

In an embodiment, the feedback information identifies a specific host orpathway to which files are to be uploaded by the user device. Forexample, assuming that speed is the factor being optimized and the userdevice has a choice of three pathways A, B and C for uploading files,the host server can determine that pathway A is the fastest of theavailable options for upload, and can thus recommend pathway A (e.g., IPaddress or a URL such as upload.xyz.com) for use by the user device inuploading files. In this example, the upload module 618 can upload thefiles to pathway A recommended by the host server.

In an embodiment, the feedback information does not identify a specificpathway to upload to, but includes weights or weighting information thatthe pathway selector 626 of the client-side feedback loop optimizer 622can use to select a pathway to upload to. For example, using a webapplication, the user device 602 can send a request to get an uploadhost suggestion from the feedback loop of the host server. Assuming thatspeed is the factor being optimized, and for a given combination of IPaddress/browser/OS factors, the user device has a choice of threepathways A, B and C for uploading files. The host server can thenprovide upload speeds clocked at pathways A, B and C as a response tothe request from the user device. An example of the response from thehost server can be substantially similar to: “A: 7284, B: 3905, C: 5932”where the speeds are in kbps. The pathway selector 626, in an exampleimplementation, can normalize, square and round off the upload speeds atthe nodes to obtain weights for the array of pathways substantiallysimilar to: “A: 3, B: 1, C: 2”. The pathway selector 626 can then selectpathway A for uploading files 3 out of 6 times (sum of 3, 1 and 2),select pathway B 1 out of 6 times and pathway C 2 out of 6 times. Theupload module 618 can then upload the files through the selectedpathways.

In an embodiment, instead of speed, an array of pathways with weightsmay be received as feedback information by the feedback informationreceiver 624. For example, the feedback information may appearsubstantially similar to:

upload.xyz.com=>7fupload1.xyz.com=>1fupload-ord.xyz.com=>2

The pathway selector 626 can use the random weighted selection methoddescribed with reference to the weighted random choice-based selectionengine 570 of the server-side feedback loop optimizer 550 in FIG. 5A.Using the random weighted selection method, the pathway selector 626 canselect the host upload.xyz.com to upload most of the files (7², or 49out of 7²+1²+2², or 54). Similarly, the pathway selector 626 can selectthe host fupload-ord.xyz.com for uploading some of the files (2², or 4out of 54), and host fupload1.xyz.com for uploading at least one file(1², or 1 out of 54).

In an embodiment, when a web application is being used to upload files,the JavaScript in the web application can include an array of hosts withweights instead of a single host. The browser engine 606 can execute theJavaScript to select a host according to the weights to upload a file.For example, on each upload, the browser engine 606 can perform anoperation substantially similar to the example below to select a hostaccording to the weights for upload:

rand(0 to 1)*(sum of weights)

The array and/or the weights can have an associated expiration timestamp(e.g., TTL value) and a new set of hosts and corresponding weights canbe fetched from the host server before an upload when the array ofhosts/weights expires.

In an embodiment, instead of using feedback information from the hostserver to determine a pathway to upload to, the user device can useclient-side feedback loop optimizer to select a pathway to enhanceupload performance. The client-side feedback loop optimizer 622 canobtain client-side feedback information by conducting a synthetic uploadspeed test via a synthetic upload speed test module 646. The syntheticupload speed test module 646 can include a connection diagnostic tool(e.g., tools similar to those provided by speedtest.net, Comscore,Akamai) that generates random data (dummy data) and uploads the dummydata to an array of hosts including a server that is local (or ingeographically proximity) to the user device to determine upload speedsassociated with the array of hosts. Uploading of dummy data to a localserver allows the synthetic upload speed test module 646 to determinethe theoretical maximum upload speed for the user device and/orconnection. The synthetic upload speed test can be run in the backgroundwhen triggered. The synthetic upload speed test module 646 may bebrowser-based or may be part of an application or client installed onthe user device.

A synthetic upload speed test configuration module 644 can determinewhen to initiate a synthetic upload speed test. In embodiments where thepathway selector 626 relies only on client-side feedback information,the synthetic upload speed test module 646 may perform speed testsperiodically and cache the results for some time for reuse, or mayperform a speed test before each upload and provide the results to thepathway selector 626 to select a pathway that optimizes uploadperformance. In other embodiments, the synthetic upload speed testconfiguration module 644 may be configured to trigger the syntheticupload speed test module 646 to obtain upload speed data associated withuploads to the array of hosts when the host server does not have enoughdata for a given combination of characteristics to compute arecommendation. The upload speed measured by the synthetic upload speedtest module 646 can then be reported to the host server via the uploadspeed reporter 648. The upload speeds reported by the user device can beaggregated by the host server. For example, the host server can insertthe upload speed measured client-side into the memcache and/or analyticsdatastore along with the upload speed measured server-side. Theclient-side reported upload speed may have a connection diagnostic speedtest or other identifier for an upload speed type. The host server canuse both types of upload speeds (i.e., measured by the host server andmeasured by the user device) to determine recommendations.

In an embodiment, the pathway selector 626 can determine the best orfastest pathway for upload based on the synthetic upload speed testresults from the synthetic upload speed test module 646. For example,synthetic upload speed test results may indicate that pathway A has anupload speed of 3000 KB/s, pathway B has an upload speed of 1000 KB/sand pathway C has an upload speed of 100 KB/s, therefore the pathwayselector 626 can use a weighted random selection method or any suitablemethod to select the best or fastest pathway for upload. In thisexample, the pathway selector 626 can select pathway A with aprobability of approximately 0.9, pathway B with a probability ofapproximately 0.1 and pathway C with the lowest probability ofapproximately 0.001, such that most of the uploads from the user devicego through the fastest or faster pathways, thereby enhancing the uploadperformance. Once an optimal pathway is selected for upload, the uploadmanager 614 containing the upload module 618 can upload the file to theselected pathway.

In an embodiment, the client-side feedback loop optimizer 622 includes ahost reachability test module (not shown) that can detect or determineif an upload/download pathway is unreachable. The host reachabilitymodule may detect the unavailability of upload/download pathways whenattempting to connect to the pathways, performing a speed test, based onerror messages (e.g., server unavailable message), and the like. When anupload/download pathway is determined to be unreachable or unavailable,the host reachability test module can report this information to thehost server. The host server, on receiving the report, can execute thepathway selection engine 566 to determine a new pathway or weights, withthe unreachable or unavailable host removed from the set of eligiblehosts. For example, if the CDN pathway is down, the host server canselect a suitable pathway from remaining options available (e.g., thedirect pathway and edge servers) for upload. This feedback informationregarding unavailability of hosts ensures reliable uploads/downloads andpreserves user experience in instances where users are behindmisconfigured or aggressive firewalls, or where there is a problem withan edge server or CDN server, or even when the host server is havingdifficulties. In some instances, detection of the unavailability orunreachability of a host can be a trigger for the client-side feedbackloop optimizer to fetch new weights (taking into account the host thatis unavailable) to update the local cache. In some embodiments, thefunctions of the host reachability test module may be performedserver-side by the host server (e.g., host server 500).

In addition to web uploads/downloads, the feedback loop optimizationtechniques can also be applied to uploads/downloads using sync clients,APIs, FTP, and other communication methods. For example, anyapplications (e.g., mobile applications) on the user device (e.g.,mobile device) having files to be uploaded to the host server can useAPI requests to directly request recommendation for the best host or thefastest host from the host server and upload to the recommended host.

FIG. 7A is a block diagram 7A00 depicting example components of a syncclient that can use feedback information to select a pathway to optimizeor enhance upload performance by dynamically shaping network traffic. Asan option, one or more instances of block diagram 7A00 or any aspectthereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the blockdiagram 7A00 or any aspect thereof may be implemented in any desiredenvironment. This figure illustrates a block diagram depicting examplecomponents of a sync client 700 running on a user device (e.g., userdevice 102 of FIG. 1A1 or user device 602 of FIG. 6A) that synchronizescopies of work items stored on the user device with copies of work itemsstored in a sync server 730, sync folder and/or a host server (e.g.,host farm 101 of FIG. 1A1) within a cloud-based collaborationenvironment. Example components of the sync server 730 are illustratedin FIG. 7B.

The sync client 700 can include, for example, a conflicts manager 704, atriggering event module 710, a copying manager 712, a state module 714,and/or a state database 718. The conflicts manager 704 can include arules engine 706 and/or an error notification module 708. The syncclient 700 can also include a sync feedback loop optimizer 716 (e.g.,upload route via client-side feedback loop optimizer 622 in FIG. 6A).Additional or fewer components/modules/engines can be included in thesync client 700 and each illustrated component.

One embodiment of the sync client 700 includes the triggering eventmodule 710 which determines when synchronization of folders shouldoccur. There are two types of triggering events for synchronization. Thefirst type of triggering event occurs when a change has been made to theserver sync folder. As a result of this event, a notification is sentfrom the notification server 150 to the triggering event module 710. Insome instances, when a user has an application open and edits a file inthe server sync folder, editing of the file causes the notificationserver 150 to send a notification to the triggering event module 710,causing the change to be downloaded to the local sync folders of othercollaborators as part of the synchronization function. The download canoccur via a pathway that provides the best download performance for agiven set of characteristics associated with the user device and/orconnection of the collaborators of the folders. In some instances, thenotification is sent to the triggering event module 710 after the userhas saved the file and closed the application.

The notification server 150 can provide real time or near real-timenotifications of activities that occur in a particular server syncfolder. In one embodiment, the triggering event module 710 can subscribeto a real-time notification channel provided by the notification server150 for a particular server sync folder to receive the notifications.

In one embodiment, the notifications provided by the notification server150 inform the triggering event module 710 that a change has occurred inthe server sync folder. In this case, the state module 714 requests fromthe current state manager 732 in the sync server 730 the current stateof the folder/file tree for the server sync folder that the local syncfolder is synchronized to.

The state module 714 also accesses the last known state of thefolder/file tree stored in the state database 718 and compares thecurrent state with the last known state to determine which file and/orfolder has changed. Once the changed files and/or folders have beenidentified, the copying manager 712 downloads the changed file(s) fromthe server sync folder to the local sync folder. The downloading can bethrough a pathway recommended by the host server for a given set ofcharacteristics or determined by the sync feedback loop optimizer 716.The second type of triggering event occurs when a change has been madeto a local sync folder on a collaborator's computer or user device. Inone embodiment, a Windows operating system of the collaborator'scomputer provides file/folder monitoring on the computer and notifiesthe triggering event module 710. Other operating systems or programsrunning on collaborators' computer systems can provide a similar type ofnotification to the triggering event module 710. Once the triggeringevent module 710 has been notified of the change to the local syncfolder, a notification is sent to the sync server 730.

When the second type of triggering event occurs, the copying manager 712uploads the changed file to replace the copy of the file stored in theserver sync folder. The changed file can be uploaded through a pathwaythat is recommended by the host server and/or the user device based onfeedback information for a set of characteristics that are applicable tothe user device (e.g., via the sync feedback loop optimizer 716). Oncethe file has been uploaded to the server sync folder, the local copy ofthe file stored on the computers of other collaborators in the workspacewhom have enabled the synchronization function are updated in a similarmanner as described above for the first type of triggering event.

An embodiment of the sync client 700 includes the conflicts manager 704which can identify when a conflict has occurred (i.e., a file or workitem has been changed at both the server sync folder and the local syncfolder) and determine how to resolve the conflict. If the triggeringevent module 710 receives a notification from the notification server150 and the local file/folder monitoring system of the changes, theconflicts manager 704 identifies the changes made to the file/folder ateach location. In one embodiment, the conflicts manager 704 calls therules engine 706 to determine what action to take to resolve theconflict. The rules engine 706 stores rules for resolving conflicts.Rules are predefined but can be changed without changing the softwareimplementing the rules engine. The rules engine 706 takes as input thetypes of changes that have occurred at the various synchronized folderssuch as edits to a work item, renaming of a work item, or moving of awork item to a different location. Then the rules engine 706 providesthe action to be performed for the particular conflict.

There are two types of conflicts: a soft conflict and a hard conflict. Ahard conflict occurs when the same operation occurs on both copies ofthe file, and a soft conflict occurs when a different operation occurson each of the two copies of the file. In the case of a hard conflict,for example, when copies of a work item have been changed at the serversync folder and at a local sync folder, the conflicts manager 704 is notable to merge the changed files. In one embodiment, the conflictsmanager 704 makes a copy of the changed work item in the local syncfolder and renames the copy with the original file name and anidentifier of the collaborator associated with the local sync folder.Next, the conflicts manager 704 downloads (e.g., via an optimal pathwaysuggested or determined by the sync feedback loop optimizer 716) thechanged work item from the server sync workspace to the local syncfolder, and then uploads (e.g., via an optimal pathway suggested ordetermined by the sync feedback loop optimizer 716) the copy of the workitem with the modified file name to the server sync folder. Thus, twoversions of the file are stored at the server sync folder and the localsync folder. Then, the error notification module 708 sends a message tothe user to notify him that the changes in his version of the work itemwere not accepted but were uploaded to the server sync folder as a newversion of the file with a new file name and requests the user to mergethe two files manually. In the case of a soft conflict, for example,when a file is moved on the server and edited locally, the conflictsmanager 704 can merge these two changes so that the file is movedlocally to the new location and the local edits are uploaded to theserver copy of the file.

In some embodiments, the sync server 730 may also include a feedbackloop optimizer performing functions similar to the feedback loopoptimizer 550 in FIG. 5B.

FIG. 8 is a flow diagram 800 depicting a method implemented by a hostserver for recommending an upload pathway for a given set ofcharacteristics to achieve enhanced upload performance by dynamicallyshaping network traffic. As an option, one or more instances of flowdiagram 800 or any aspect thereof may be implemented in the context ofthe architecture and functionality of the embodiments described herein.Also, the flow diagram 800 or any aspect thereof may be implemented inany desired environment. This figure illustrates an example flow diagramdepicting a method implemented by a host server 804 (e.g., host farm 101of FIG. 1A1; host server 500 of FIG. 5A) for recommending an uploadpathway for a given set of characteristics for a user device 802 (e.g.,user device 102 of FIG. 1A1 and user device 102 FIG. 1A3) to upload toenhance upload performance.

In this specific example flow diagram, the shown user device 802 sendsan upload start request 812 to the shown host server 804. The uploadstart request 812 can be a request that serves the dual purpose ofinforming the host server 804 regarding an upload about to start andrequesting a recommendation for an upload pathway to upload a file. Insome implementations, instead of the upload start request 812, arecommendation request can be made. The upload start request 812 can bean HTTP request (e.g., an HTTP GET/POST request or an API request) orsimilar request and can include attributes or characteristics of theuser device 802. Example characteristics can include a complete orpartial IP address (IPv4 or IPv6 address formats), browser name/version,OS name/version, file type, file size, user ID, account type, and thelike. The host server 804 receives the upload start request 812 andextracts the user device characteristics from the request to identify akey at block 814. The key corresponds to a set of characteristics suchas “01.102.103.XXX, Windows XP SP2, Firefox 4.0”. The host server 804,at block 816, determines whether there is sufficient data aggregated toselect an upload pathway for the identified key. In someimplementations, there may not be sufficient data aggregated todetermine that one upload pathway is definitively better or faster thananother. For example, upload pathway 1 may have an aggregate speed of 2MB/s but upload pathways 2 and 3 may have no aggregate speed values. Insuch an instance, the host server 804 can select an upload pathway withequal probability or round robin method at block 818. The host server804 can then send a response 822 to the upload start request 812. Theresponse 822 includes an identification of an upload pathway selectedwithout any optimization, using equal probability method. The response822 can include information such as a session ID, selected uploadpathway's URL, IP address, or other identifier or address. In theillustrated example, the host server 804 selected upload pathway 808(e.g., host server 804). The user device 802 then uploads the data files824 to the upload pathway 808, which is the host server 804 in thisparticular example.

Alternately, at block 816, if the host server 804 has sufficient dataaggregated to recommend an upload pathway, the host server can selectthe recommended upload pathway and provide information regarding therecommended upload pathway to the user device 802 in the response 822.Once again, the user device can then upload a file to the recommendedupload pathway (pathway 808 in this example). If the selected hostpathway 808 is not the host server 804 (i.e., not a direct upload), buta CDN node for example, in such an instance the CDN node can receive theuploaded files. The CDN node can perform a network address translation826 to route the uploaded data files 828 to the host server 804. Inother words, the upload from the user device is initially pointed to theIP address of the CDN node, and the CDN node, on receiving a datapacket, rewrites the source IP address and destination IP address toforward the data packet to the host server 804. Since the IP address ofthe user device 802 can get wiped during network address translation,the session ID can be used to determine the IP address associated withdata files 828 received at the host server 804.

The host server 804 upon receiving the data files can calculate theupload speed at block 830 based on the timestamps corresponding toreceipt of a first byte and a last byte. In some instances, thecalculated upload speed can be unusable or an outlier. The host server804 can filter out the unusable/outlier upload speed at block 832. Atblock 834, the host server 804 can update the aggregate upload speed forthe selected upload pathway and the key in a memcache. Further at block836, all the data associated with the upload is stored in associationwith the key in an analytics datastore.

FIG. 9A is a logic flow diagram 9A00 depicting a method for selecting apathway from an array of pathways based on a set of weights to enhanceupload/download performance. As an option, one or more instances oflogic flow diagram 9A00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the logic flow diagram 9A00 or any aspectthereof may be implemented in any desired environment.

The method starts at block 902. At block 904, a user device detects acollaborator's request to upload a file or files. In some instances, therequest to upload a file or files may be automatically generated (e.g.,via sync client 700 for syncing files on the user device with those inthe host server). The user device sends a request to a host server(e.g., host farm 101 in FIG. 1A1 or host server 500 in FIG. 5A) forinformation regarding upload pathways at block 906. The informationregarding upload pathways can be used by the user device to select anupload pathway from the group to upload a file. In some implementations,the user device can request the host server to recommend an uploadpathway to which the user device should upload a file. The request caninclude information such as an IPv4 or IPv6 address, browser version, OSversion, user ID, file size, file type, and the like. In someimplementations, the request can also specify one or more factors to beoptimized (e.g., upload speed, upload speed for file size or type, andthe like). Alternately, the host server can determine which optimizationfactor or factors are considered in recommending an upload pathway or anarray of upload pathways for upload.

At block 908, the user device receives a session ID and a set of weightscorresponding to an array of pathways from the host server as feedbackinformation. The user device then randomly selects a pathway based onthe set of weights at block 910. In some instances, the selection of thepathway can be made server-side. In an embodiment, the decision toimplement the selection server-side or client-side may be based onvarious factors such as security, avoiding caching on the user device,use of most current information, and the like. The user device thenuploads a file to the selected pathway at block 912. If there aremultiple files to be uploaded as determined at block 914, the userdevice performs another random selection operation to select a pathwaybased on the set of weights at block 910. Via the random weightedselection approach, the user device achieves upload performanceimprovement by selecting a pathway that is faster more frequently than apathway that is slower. At the same time, since the user device can endup trying all the pathway options, any change in the upload speed of anypathways can be detected and taken into account by the host server suchthat a different set of weights can be provided to the user device forthe next upload.

FIG. 9B is a logic flow diagram 9B00 depicting a method for selecting apathway from an array of pathways based on a set of weights cached in auser device. As an option, one or more instances of logic flow diagram9B00 or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the logic flow diagram 9B00 or any aspect thereof may beimplemented in any desired environment.

The method starts at block 920. A user device (e.g., user device 102 ofFIG. 1A1; user device 402 of FIG. 4, user device 602 of FIG. 6A, userdevice 802 of FIG. 8, etc.) detects a user request to upload a file orfiles at block 922. At block 924, the user device determines if theweights for an array of pathways are available in a local cache of theuser device. In one implementation, the user device, when looking forweights in the local cache can look for weights corresponding to a setof user device characteristics. In some instances, it is possible for auser to use his or her user device at home (e.g., IP address123.12.13.XX) and at a coffee shop in another part of the town (e.g., IPaddress 123.14.15.XX). Since the IP octet part of the set ofcharacteristics can be different for the same user device, the pathwaysand the corresponding weights for upload from home can be different fromthe weights for upload from another location. Consequently,consideration of the “home” set of characteristics can lead to aselection of pathway different from consideration of the “office” set ofcharacteristics. If the weights are not available in the local cache,the user device can fetch new weights for an array of pathways at block928. However, if weights are available, the user device can determinewhether the weights in cache are expired at block 926. If the weightsare expired, the user device can fetch new weights for an array ofpathways for a set of user device characteristics at block 928. The userdevice then stores the fetched weights in the local cache, replacing theexpired weights at block 930.

At block 932, the user device using the new weights fetched from thehost server or the weights in the cache that are still usable randomlyselects a pathway from an array of pathways. At block 934, the userdevice uploads a file to the selected pathway. If more than one file isto be uploaded during the same session as determined at block 936, theuser device repeats the process at block 932. When there are no otherfiles to be uploaded, the process terminates at block 940.

FIG. 10 is a logic flow diagram 1000 depicting a method for selecting anoptimal upload pathway by a user device based on client-side feedbackinformation. As an option, one or more instances of logic flow diagram1000 or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the logic flow diagram 1000 or any aspect thereof may beimplemented in any desired environment.

In an embodiment, the client-side feedback information for selecting anoptimal pathway for an upload/download can be aggregated by performingsynthetic speed tests (web-based or via a local client on a userdevice). A synthetic speed test may start as a background process atblock 1002. At block 1004, the user device sends dummy data to an edgeserver located nearby. At block 1006, the user device sends dummy datadirectly to a host server (e.g., the host farm 101 in FIG. 1A1; hostserver 500 in FIG. 5A). Similarly, at block 1008 and 1010, the userdevice sends dummy data to a CDN node and a local server located ingeographic proximity to the user device, respectively. The dummy datathat is sent to these pathways are the same size/type and may be sent atapproximately the same time. At block 1012, the user device determinesupload speed through each of the pathways based on the total size of thedata uploaded and duration of the upload. The upload speed to the localserver can be taken as the theoretical maximum upload speed possible forthe given connection. The upload speeds at the edge server, the hostserver and the CDN node will generally be lower than the upload speed atthe local server. The theoretical maximum upload speed can thus be usedto identify upload speeds measured at other pathways that may beoutliers or erroneous. In one implementation, the upload speed that iscalculated at block 1012 can be an aggregate or average speed (e.g.,test can be repeated a number of times and an average of the results maybe considered). Alternately, other methodology for determiningupload/download speeds can be used. For example, the upload speed (orconnection speed) can be first estimated by sending dummy data to apathway. Based on the estimated speed, appropriately sized chunks can bepushed to the pathway. The chunks can be sorted by speed and speedscorresponding to a range (e.g., 30th percentile to 90th percentile) canbe selected and averaged to determine the upload speed.

At block 1014, the user device receives a user request (or an automatedrequest) to initiate an upload. In an embodiment, the user device candetect throttling or bandwidth capping. When the speeds across multiplepathways are similar, the user's Internet connection is likely subjectto upload throttling or bandwidth capping, causing the uploads to belimited to the same speed, regardless of the pathway selected. At block1016, if the user device detects upload throttling, the user device canselect an upload pathway using round robin method at block 1020.Alternately, if there is no upload throttling, the user device canselect an upload pathway that has the fastest aggregate upload speed atblock 1018. In some implementations, the user device can also select anupload pathway based on weights determined from the upload speeds fromthe upload pathways. At block 1022, the user device uploads data to theselected upload pathway. Further, at block 1024, the user device canreport the aggregate upload speeds determined for each upload pathway tothe host server.

FIG. 11 is a logic flow diagram 1100 depicting a method for usingfeedback information to optimize data transfer performance. As anoption, one or more instances of logic flow diagram 1100 or any aspectthereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the logic flowdiagram 1100 or any aspect thereof may be implemented in any desiredenvironment.

In an embodiment, a user device and/or a host server can detect that adata transfer event is about to occur at block 1105. The data transferevent may be an upload event or a download event. Based on a set ofcharacteristics associated with the data transfer event, the user deviceand/or the host server can select a host from a group of hosts as apathway to transfer the data to optimize data transfer performance atblock 1110. At block 1115, the data associated with the data transferevent transferred to the selected host can be received at the hostserver and/or the user device. At block 1120, a speed with which thedata associated with the data transfer event is transferred through theselected host is determined by the host server and/or the user device.An aggregation profile corresponding to the set of characteristicsassociated with the data transfer event can be updated at block 1125.The aggregation profile may be stored at the host server and/or the userdevice.

FIG. 12 is a logic flow diagram 1200 depicting a method for optimizingfile transfer performance to improve user experience. As an option, oneor more instances of logic flow diagram 1200 or any aspect thereof maybe implemented in the context of the architecture and functionality ofthe embodiments described herein. Also, the logic flow diagram 1200 orany aspect thereof may be implemented in any desired environment.

In an embodiment, a request for a host recommendation to transfer a fileis received by a host server (e.g., host farm 101 in FIG. 1A1; hostserver 500 in FIG. 5A) at block 1205. At block 1210, a host server canuse the combination of attributes as a key to determine aggregate speedinformation of an array of hosts. At block 1215, the host server canfurther use the aggregate speed information relating to the array ofhosts to select a host to recommend for the transfer of the file. Forexample, the host server can select a host that has the fastest speedamong the array of hosts. At block 1220, the host server can provide theselected host to a client device for use in transferring the file. Theclient device can use the selected host to upload the file. At block1225, the host server can receive the file uploaded to the selectedhost. For example, when the selected host is a CDN server or an edgeserver, the uploaded file is forwarded by the CDN server or the edgeserver to the host server. When the selected host is the host server,the host server can receive the file directly. At block 1230, the hostserver measures an upload speed for the upload of the file. For example,the host server can measure the duration between receiving of the firstbyte and the last byte of the file and the size of the file to determinethe upload speed. At block 1230, the host server can update the selectedhost's aggregate speed information for the combination of attributes.For example, the host server can log all of the data relating to theupload in an analytics datastore, and use the upload speed informationto update the aggregate speed and upload count associated with thecorresponding host and combination of attributes.

FIG. 13 is a logic flow diagram 1300 depicting a method for enhancingthe upload performance. As an option, one or more instances of logicflow diagram 1300 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the logic flow diagram 1300 or any aspectthereof may be implemented in any desired environment.

In an embodiment, a host server (e.g., host farm 101 in FIG. 1A1; hostserver 500 in FIG. 5A), observes the upload of files through multiplepathways at block 1305. Based on the observation, the host serveridentifies a pathway from the multiple pathways that is fastest (or mostreliable, best, etc.) for a given set of characteristics at block 1310.At block 1315, the host server aggregates some of information from theobservation into groups defined by a set of characteristics and storesthe aggregated information in a memcache. The memcache is periodicallyflushed to remove information that is out of date at block 1320. Atblock 1325, host server can aggregate all of the information from theobservation in a persistent datastore. In the case of an accidentalflush of the memcache, before the information in the memcache is out ofdate, the host server can retrieve information from the persistentdatastore to repopulate the memcache.

FIG. 14 is a chart 1400 depicting uploads by pathways over time asgenerated using information tracked or observed at the host server. Asan option, one or more instances of chart 1400 or any aspect thereof maybe implemented in the context of the architecture and functionality ofthe embodiments described herein. Also, the chart 1400 or any aspectthereof may be implemented in any desired environment. The graphicaldiagram in this figure depicts uploads by pathways over time generatedusing the information tracked or observed by the upload analytics engine532. As depicted, depending on the time, the number of uploads throughpathways varies. At 8 pm, a large number of uploads are through CDNservers 1410, while a smaller number of uploads are through otherpathways. Just after 8 am, a larger number of uploads are throughpathway 1425 (e.g., Washington D.C. edge server). This type ofobservational data can assist in determining how effective pathways arein accelerating uploads (or downloads) at various times of day.

The administrator user interface 536 allows the admin to specify,adjust, add and/or delete various parameters and settings of the uploadpathway feedback loop optimizer. For example, via the admin userinterface, types of upload data to be logged can be modified. The adminuser interface can also be used to perform the example analyses via theupload data analysis module 534.

FIG. 15 is a system 1500 using pathway feedback information to optimizeor enhance full-duplex upload or download performance. As an option, oneor more instances of system 1500 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the system 1500 or any aspectthereof may be implemented in any desired environment.

Additional Embodiments of the Disclosure Additional PracticalApplication Examples

FIG. 16 depicts a system 1600 as an arrangement of computing modulesthat are interconnected so as to operate cooperatively to implementcertain of the herein-disclosed embodiments. The partitioning of system1600 is merely illustrative and other partitions are possible. As anoption, the present system 1600 may be implemented in the context of thearchitecture and functionality of the embodiments described herein. Ofcourse, however, the system 1600 or any operation therein may be carriedout in any desired environment. The system 1600 comprises at least oneprocessor and at least one memory, the memory serving to store programinstructions corresponding to the operations of the system. As shown, anoperation can be implemented in whole or in part using programinstructions accessible by a module. The modules are connected to acommunication path 1605, and any operation can communicate with otheroperations over communication path 1605. The modules of the system can,individually or in combination, perform method operations within system1600. Any operations performed within system 1600 may be performed inany order unless as may be specified in the claims. The shown embodiment16 implements a portion of a computer system, shown as system 1600,comprising a computer processor to execute a set of program codeinstructions (see module 1610) and modules for accessing memory to holdprogram code instructions to perform: opening a client-server sessionconfigurable to establish a full-duplex persistent network communicationsocket between a client device and a server (see module 1620);receiving, by the server, an indication to upload a file or object overthe full-duplex persistent network communication socket from a locationaccessible by the client device to one or more storage locationsaccessible by the server (see module 1630); allocating resources to openthe full-duplex persistent network communication socket (see module1640); receiving, from the client device, an initial portion of a datatransfer stream comprising a succession of portions of the file orobject (see module 1650); selecting a first host from a group of hoststo use for transferring at least some of the succession of portions ofthe file or object (see module 1660); receiving, from at least one ofthe group of hosts, a set of performance characteristics pertaining tothe data transfer stream, wherein the a set of performancecharacteristics is responsive to the at least some portion of successionof portions of the file or object (see module 1670); and sending anaspect of the set of performance characteristics pertaining to the datatransfer stream to determine a client-side shaping parameter (see module1680).

System Architecture Overview Additional System Architecture Examples

FIG. 17 depicts an exemplary architecture of components suitable forimplementing embodiments of the present disclosure, and/or for use inthe herein-described environments. Computer system 1700 includes a bus1706 or other communication mechanism for communicating information. Thebus interconnects subsystems and devices such as a CPU, or a multi-coreCPU (e.g., data processors 1707), a system memory (e.g., main memory1708, or an area of random access memory RAM), a non-volatile storagedevice or area (e.g., ROM 1709), an internal or external storage device1710 (e.g., magnetic or optical), a data interface 1733, acommunications interface 1714 (e.g., PHY, MAC, Ethernet interface,modem, etc.). The aforementioned components are shown within processingelement partition 1701, however other partitions are possible. The showncomputer system 1700 further comprises a display 1711 (e.g., CRT orLCD), various input devices 1712 (e.g., keyboard, cursor control), andan external data repository 1731.

According to an embodiment of the disclosure, computer system 1700performs specific operations by processor 1707 executing one or moresequences of one or more program code instructions contained in amemory. Such instructions (e.g., program instructions 1702 ₁, programinstructions 1702 ₂, program instructions 1702 ₃, etc.) can be containedin or can be read into a storage location or memory from any computerreadable/usable medium such as a static storage device or a disk drive.The sequences can be organized to be accessed by one or more processingentities configured to execute a single process or configured to executemultiple concurrent processes to perform work. A processing entity canbe hardware-based (e.g., involving one or more cores) or software-based,and/or can be formed using a combination of hardware and software thatimplements logic, and/or can carry out computations and/or processingsteps using one or more processes and/or one or more tasks and/or one ormore threads or any combination therefrom.

According to an embodiment of the disclosure, computer system 1700performs specific networking operations using one or more instances ofcommunications interface 1714. Instances of the communications interface1714 may comprise one or more networking ports that are configurable(e.g., pertaining to speed, protocol, physical layer characteristics,media access characteristics, etc.) and any particular instance of thecommunications interface 1714 or port thereto can be configureddifferently from any other particular instance. Portions of acommunication protocol can be carried out in whole or in part by anyinstance of the communications interface 1714, and data (e.g., packets,data structures, bit fields, etc.) can be positioned in storagelocations within communications interface 1714, or within system memory,and such data can be accessed (e.g., using random access addressing, orusing direct memory access DMA, etc.) by devices such as processor 1707.

The communications link 1715 can be configured to transmit (e.g., send,receive, signal, etc.) communications packets 1738 comprising anyorganization of data items. The data items can comprise a payload dataarea 1737, a destination address 1736 (e.g., a destination IP address),a source address 1735 (e.g., a source IP address), and can includevarious encodings or formatting of bit fields to populate the shownpacket characteristics 1734. In some cases the packet characteristicsinclude a version identifier, a packet or payload length, a trafficclass, a flow label, etc. In some cases the payload data area 1737comprises a data structure that is encoded and/or formatted to fit intobyte or word boundaries of the packet.

In some embodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement aspects of thedisclosure. Thus, embodiments of the disclosure are not limited to anyspecific combination of hardware circuitry and/or software. Inembodiments, the term “logic” shall mean any combination of software orhardware that is used to implement all or part of the disclosure.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 1707 for execution. Such a medium may take many formsincluding, but not limited to, non-volatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks suchas disk drives or tape drives. Volatile media includes dynamic memorysuch as a random access memory.

Common forms of computer readable media includes, for example, floppydisk, flexible disk, hard disk, magnetic tape, or any other magneticmedium; CD-ROM or any other optical medium; punch cards, paper tape, orany other physical medium with patterns of holes; RAM, PROM, EPROM,FLASH-EPROM, or any other memory chip or cartridge, or any othernon-transitory computer readable medium. Such data can be stored, forexample, in any form of external data repository 1731, which in turn canbe formatted into any one or more storage areas, and which can compriseparameterized storage 1739 accessible by a key (e.g., filename, tablename, block address, offset address, etc.).

Execution of the sequences of instructions to practice certainembodiments of the disclosure are performed by a single instance of thecomputer system 1700. According to certain embodiments of thedisclosure, two or more instances of computer system 1700 coupled by acommunications link 1715 (e.g., LAN, PTSN, or wireless network) mayperform the sequence of instructions required to practice embodiments ofthe disclosure using two or more instances of components of computersystem 1700.

The computer system 1700 may transmit and receive messages such as dataand/or instructions organized into a data structure (e.g.,communications packets 1738). The data structure can include programinstructions (e.g., application code 1703), communicated throughcommunications link 1715 and communications interface 1714. Receivedprogram code may be executed by processor 1707 as it is received and/orstored in the shown storage device or in or upon any other non-volatilestorage for later execution. Computer system 1700 may communicatethrough a data interface 1733 to a database 1732 on an external datarepository 1731. Data items in a database can be accessed using aprimary key (e.g., a relational database primary key).

The partitioning shown is merely one sample partition. Other partitionscan include multiple data processors, and/or multiple communicationsinterfaces, and/or multiple storage devices, etc. within a partition.For example, a partition can bound a multi-core processor (e.g.,possibly including embedded or co-located memory), or a partition canbound a computing cluster having plurality of computing elements, any ofwhich computing elements are connected directly or indirectly to acommunications link. A first partition can be configured to communicateto a second partition. A particular first partition and particularsecond partition can be congruent (e.g., in a processing element array)or can be different (e.g., comprising disjoint sets of components).

A module as used herein can be implemented using any mix of any portionsof the system memory and any extent of hard-wired circuitry includinghard-wired circuitry embodied as a processor 1707. Some embodimentsinclude one or more special-purpose hardware components (e.g., powercontrol, logic, sensors, transducers, etc.). A module may include one ormore state machines and/or combinational logic used to implement orfacilitate the performance characteristics of dynamically shapingnetwork traffic when using full-duplex websocket communications.

Various implementations of the database 1732 comprise storage mediaorganized to hold a series of records or files such that individualrecords or files are accessed using a name or key (e.g., a primary keyor a combination of keys and/or query clauses). Such files or recordscan be organized into one or more data structures (e.g., data structuresused to implement or facilitate aspects of dynamically shaping networktraffic when using full-duplex websocket communications.). Such files orrecords can be brought into and/or stored in volatile or non-volatilememory.

Returning to discussion of the heretofore introduced environments, someof the environments includes components with which various of theherein-disclosed systems can be implemented. Not all of the componentsshown may be required to practice the embodiments, and variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the disclosure.

Various environment in which embodiments of the disclosure operate mayinclude local area networks (LANs)/wide area networks (WANs), wirelessnetworks, client devices (e.g., user stations). The overall network,including any sub-networks and/or wireless networks, are incommunication with, and enables communication between components in theenvironment.

Instances of client devices may include virtually any computing devicecapable of communicating over a network to send and receive information,including instant messages, performing various online activities or thelike. It should be recognized that more or fewer client devices may beincluded within a system such as described herein, and embodiments aretherefore not constrained by the number or type of client devicesemployed.

Devices that may operate as client devices may include devices that canconnect using a wired or wireless communications medium such as personalcomputers, servers, multiprocessor systems, microprocessor-based orprogrammable consumer electronics, network PCs or the like. In someembodiments, client devices may include virtually any portable computingdevice capable of connecting to another computing device and receivinginformation such as a laptop computer, a smart phone, a tablet computer,or the like. Portable or mobile computer devices are may also include oroperate in conjunction with other portable devices such as cellulartelephones, display pagers, radio frequency (RF) devices, infrared (IR)devices, personal digital assistants (PDAs), handheld computers,wearable computers integrated devices combining one or more of thepreceding devices and the like. As such, client devices can range widelyin terms of capabilities and features. Moreover, client devices mayprovide access to various computing applications including a browser orother web-based applications. A web-enabled client device may include abrowser application that is configured to receive and to send web pages,web-based messages and the like. The browser application may beconfigured to receive and display graphics, text, multimedia and thelike, employing virtually any web-based language including a wirelessapplication protocol messages (WAP) and the like. In certainembodiments, the browser application is enabled to employ handhelddevice markup language (HDML), wireless markup language (WML),WMLScript, JavaScript, standard generalized markup language (SGML),HyperText markup language (HTML), eXtensible markup language (XML) andthe like, to display and send a message. In certain embodiments, a userof the client device may employ the browser application to performvarious activities over a network (online). However, another applicationmay also be used to perform various online activities.

Client devices may include at least one client application that isconfigured to receive and/or send data between other computing devices(e.g., server components). The client application may include acapability to provide send and/or receive content or the like. Theclient application may further provide information that identifiesitself including a type, capability, name or the like. In certainembodiments, a client device may uniquely identify itself through any ofa variety of mechanisms including a phone number, mobile identificationnumber (MIN), an electronic serial number (ESN), or other mobile deviceidentifier. The information may also indicate a content format that themobile device is enabled to employ. Such information may be provided ina network packet or the like, sent between other client devices, or sentbetween other computing devices.

Client devices may be further configured to include a client applicationthat enables an end-user to log into an end-user account that may bemanaged by another computing device. Such end-user accounts, in onenon-limiting example, may be configured to enable the end-user to manageone or more online activities including, in one non-limiting example,search activities, social networking activities, browse variouswebsites, communicate with other users, participate in gaming, interactwith various applications or the like. However, participation in onlineactivities may also be performed without logging into the end-useraccount.

A wireless communication capability is configured to couple clientdevices and other components with network. Wireless network may includeany of a variety of wireless sub-networks that may further overlaystand-alone and/or ad-hoc networks and the like, to provide aninfrastructure-oriented connection for client devices. Such sub-networksmay include mesh networks, wireless LAN (WLAN) networks, cellularnetworks and the like. In certain embodiments, the system may includemore than one wireless network.

A wireless network may further include an autonomous system ofterminals, gateways, routers, mobile network edge devices and the likewhich may be connected by wireless radio links, etc. Connections may beconfigured to move freely and randomly and organize themselvesarbitrarily such that the topology of a wireless network may changerapidly. A wireless network may further employ a plurality of accesstechnologies including AMPS and/or second generation (2G), and/or thirdgeneration (3G), and/or fourth generation (4G) generation radio accessfor cellular systems, WLAN, wireless router (WR) mesh and the like. Theforegoing access technologies as well as emerging and/or future accesstechnologies may enable wide area coverage for mobile devices such asclient devices with various degrees of mobility. In one non-limitingexample, wireless network may enable a radio connection through a radionetwork access such as a global system for mobile (GSM) communication,general packet radio services (GPRS), enhanced data GSM environment(EDGE), wideband code division multiple access (WCDMA) and the like. Awireless network may include any wireless communication mechanism bywhich information may travel between client devices and/or between anyother computing devices and/or over or between other networks or networkcomponents.

Any of the foregoing networks can be configured to couple networkdevices with other computing devices and communication can includecommunicating over the Internet. In some situations communication iscarried out using combinations of LANs, WANs, as well as directconnections such as through a universal serial bus (USB) port, otherforms of computer readable media. On an interconnected set of LANs,including those based on differing architectures and protocols, a routeracts as a link between LANs, enabling messages to be sent from one toanother. In addition, communications links within LANs may includetwisted wire pair or coaxial cable, while communications links betweennetworks may use analog telephone lines, full or fractional dedicateddigital lines including T1, T2, T3, and T4, and/or other carriermechanisms including, for example, E-carriers, integrated servicesdigital networks (ISDNs), digital subscriber lines (DSLs), wirelesslinks including satellite links, or other communications links known tothose skilled in the art. Moreover, communications links may furtheremploy any of a variety of digital signaling technologies including,without limit, for example, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12,OC-48 or the like. Furthermore, remote computers and other relatedelectronic devices can be remotely connected to either LANs or WANs viaa modem and temporary telephone link. In various embodiments, network106 may be configured to transport information of an Internet protocol(IP). In some cases, communication media carries computer readableinstructions, data structures, program modules, or other transportmechanism and includes any information delivery media. By way ofexample, communication media includes wired media such as twisted pair,coaxial cable, fiber optics, wave guides, and other wired media andwireless media such as acoustic, RF, infrared, and other wireless media.

Furthermore, the host farm 101 of the online or web-based collaborationenvironment provides platform and application independent methods andfeatures for networked file access and editing by a remote device (e.g.,by user devices 102). Specifically, the host farm 101 and componentsresiding on a client side (e.g., on a user device 102) enables a user toedit files or other work items on the host farm 101 using their ownchoice of applications, or any application that is available on thedevice 102 they are using to access/edit the file, and regardless of thedevice 102 platform (e.g., mobile, or desktop or operating system).

In one embodiment, the client devices 102 communicate with the host farm101 and/or notification server 150 over network 106. In general, network106, over which the client devices 102, the host farm 101, and/ornotification server 150 communicate, may be a cellular network, atelephonic network, an open network, such as the Internet, or a privatenetwork, such as an intranet and/or the extranet, or any combinationthereof. For example, the Internet can provide file transfer, remote login, email, news, RSS, cloud-based services, instant messaging, visualvoicemail, push mail, VoIP, and other services through any known orconvenient protocol, such as, but is not limited to the TCP/IP protocol,open system interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH,RS-232, SDH, SONET, etc.

The network 106 can be any collection of distinct networks operatingwholly or partially in conjunction to provide connectivity to the clientdevices 102 and the host farm 101 and may appear as one or more networksto the serviced systems and devices. In one embodiment, communicationsto and from the client devices 102 can be achieved by an open network,such as the Internet, or a private network, such as an intranet and/orthe extranet. In one embodiment, communications can be achieved by asecure communications protocol, such as secure sockets layer (SSL), ortransport layer security (TLS).

In addition, communications can be achieved via one or more networks,such as, but not limited to, one or more of WiMax, a local area network(LAN), wireless local area network (WLAN), a personal area network(PAN), a campus area network (CAN), a metropolitan area network (MAN), awide area network (WAN), a wireless wide area network (WWAN), enabledwith technologies such as, by way of example, global system for mobilecommunications (GSM), personal communications service (PCS), digitaladvanced mobile phone service (D-amps), Bluetooth, Wi-Fi, fixed wirelessdata, 2G, 2.5G, 3G, 4G, IMT-advanced, pre-4G, 3G LTE, 3GPP LTE, LTEadvanced, mobile WiMax, WiMax 2, wirelessMAN-advanced networks, enhanceddata rates for GSM evolution (EDGE), general packet radio service(GPRS), enhanced GPRS, iBurst, UMTS, HSPDA, HSUPA, HSPA, UMTS-TDD,1xRTT, EV-DO, messaging protocols such as, TCP/IP, SMS, MMS, extensiblemessaging and presence protocol (XMPP), real time messaging protocol(RTMP), instant messaging and presence protocol (IMPP), instantmessaging, USSD, IRC, or any other wireless data networks or messagingprotocols.

In the foregoing specification, the disclosure has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the disclosure. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the disclosure. The specification and drawingsto be regarded in an illustrative sense rather than in a restrictivesense.

What is claimed is:
 1. A method for performing data transfer in acloud-based environment, comprising: identifying a server in acloud-based environment, wherein one or more storage devices areaccessible by the server to provide cloud-based storage as a serviceacross a network; identifying a client device to communicate with theserver to access the one or more storage devices to use the cloud-basedstorage offered by the server; determining that a data transfer is to beperformed between the client device and the server; opening aclient-server session between the client device and the server, whereinfull-duplex persistent network communications are established betweenthe client device and the server; receiving, by the server, anindication to upload or download one or more files or objects over thefull-duplex persistent network communications at a location accessibleby the client device relative to one or more storage locations on theone or more storage devices accessible by the server; allocatingresources to open the full-duplex persistent network communications;transferring an initial portion of a data transfer stream comprising asuccession of portions of the one or more files or objects; and usingfeedback information to dynamically shape performance characteristics ofthe data transfer.
 2. The method of claim 1, further comprising:selecting a first host from a group of hosts to use for transferring atleast some of the succession of portions of the one or more files orobjects; determining, from at least one of the group of hosts, a set ofperformance characteristics pertaining to the data transfer stream,wherein the a set of performance characteristics is responsive to the atleast some portion of succession of portions of the one or more files orobjects; and determining a client-side shaping parameter to dynamicallyshape the performance characteristics of the data transfer based atleast in part on an aspect of the set of performance characteristics. 3.The method of claim 2, further comprising determining, from at least oneof the group of hosts, an updated set of performance characteristicspertaining to the data transfer stream.
 4. The method of claim 3,wherein the performance characteristics are updated by updating anaggregate speed or updating a data transfer count.
 5. The method ofclaim 1, further comprising sending a set of second successive ofportions after receiving a first acknowledgement frame.
 6. The method ofclaim 5, wherein a first acknowledgement frame is received over thefull-duplex persistent network communications.
 7. The method of claim 6,wherein the first acknowledgement frame comprises a set of updatedperformance characteristics of the data transfer stream.
 8. The methodof claim 1, wherein the full-duplex persistent network communications isa websocket.
 9. The method of claim 1, wherein the performancecharacteristics comprises an aggregate speed and a data transfer count.10. The method of claim 1, wherein the performance characteristicscomprises at least one of, a partial Internet Protocol (IP) address, andan operating system name associated with the client device.
 11. Themethod of claim 1, wherein the performance characteristics comprises abrowser name.
 12. The method of claim 1, wherein the data transferstream originates at the client device.
 13. The method of claim 1,wherein the data transfer stream comprises a request to upload data. 14.The method of claim 1, wherein the data transfer stream comprises arequest to download data.
 15. The method of claim 1, wherein the servercomprises at least two of, an edge server, a content delivery networkserver, and a host server.
 16. The method of claim 1, furthercomprising: selecting a host from among a group of hosts based at leastin part on a host speed.
 17. The method of claim 1, wherein the datacomprises a data upload or a data download.
 18. A computer programproduct, embodied in a non-transitory computer readable medium, thecomputer readable medium having stored thereon a sequence ofinstructions which, when executed by a processor causes the processor toexecute a process, the process comprising: identifying a server in acloud-based environment, wherein one or more storage devices areaccessible by the server to provide cloud-based storage as a serviceacross a network; identifying a client device to communicate with theserver to access the one or more storage devices to use the cloud-basedstorage offered by the server; determining that a data transfer is to beperformed between the client device and the server; opening aclient-server session between the client device and the server, whereinfull-duplex persistent network communications are established betweenthe client device and the server; receiving, by the server, anindication to upload or download one or more files or objects over thefull-duplex persistent network communications for a location accessibleby the client device relative to one or more storage locations on theone or more storage devices accessible by the server; allocatingresources to open the full-duplex persistent network communications;receiving, from the client device, an initial portion of a data transferstream comprising a succession of portions of the one or more files orobjects; and using feedback information to dynamically shape performancecharacteristics of the data transfer.
 19. The computer program productof claim 18, wherein the computer readable medium further comprisesinstructions which, when executed by a processor causes the processor toexecute the process further comprising: selecting a first host from agroup of hosts to use for transferring at least some of the successionof portions of the one or more files or objects; determining, from atleast one of the group of hosts, a set of performance characteristicspertaining to the data transfer stream, wherein the a set of performancecharacteristics is responsive to the at least some portion of successionof portions of the one or more files or objects; and determining aclient-side shaping parameter to dynamically shape the performancecharacteristics of the data transfer based at least in part on an aspectof the set of performance characteristics.
 20. The computer programproduct of claim 19, wherein the computer readable medium furthercomprises instructions for receiving, from at least one of the group ofhosts, an updated set of performance characteristics pertaining to thedata transfer stream.
 21. The computer program product 18, theperformance characteristics are updated by updating an aggregate speedor updating a data transfer count.
 22. A system comprising: a networkinterface configurable to receive a request for a client-server sessionconfigurable to establish full-duplex persistent network communicationsbetween a client device and a server in a cloud-based environment,wherein one or more storage devices are accessible by the server toprovide cloud-based storage as a service across a network, and anindication to upload one or more files or objects over the full-duplexpersistent network communications from a location accessible by theclient device to the one or more storage locations accessible by theserver; and a computing platform configurable to execute instructions toimplement: determining that a data transfer is to be performed betweenthe client device and the server; opening a client-server sessionbetween the client device and the server, wherein full-duplex persistentnetwork communications are established between the client device and theserver; receiving, by the server, an indication to upload or downloadone or more files or objects over the full-duplex persistent networkcommunications at a location accessible by the client device relative toone or more storage locations on the one or more storage devicesaccessible by the server; allocating resources to open the full-duplexpersistent network communications; receiving, from the client device, aninitial portion of a data transfer stream comprising a succession ofportions of the one or more files or objects; and using feedbackinformation to dynamically shape performance characteristics of the datatransfer.
 23. The system of claim 22, wherein the instructions furthercomprise program code to: select a first host from a group of hosts touse for transferring at least some of the succession of portions of theone or more files or objects; determine, from at least one of the groupof hosts, a set of performance characteristics pertaining to the datatransfer stream, wherein the a set of performance characteristics isresponsive to the at least some portion of succession of portions of theone or more files or objects; and determine a client-side shapingparameter to dynamically shape the performance characteristics of thedata transfer based at least in part on an aspect of the set ofperformance characteristics.
 24. The system of claim 23, wherein thenetwork interface is configurable to receive, from at least one of thegroup of hosts, an updated set of performance characteristics pertainingto the data transfer stream.